libvirtd virNetSocketReadWire:1811 : End of file while reading data: Input/output error

Status changed to 'Confirmed' because the bug affects multiple users.

Also seeing this issue during live migration with below libvirt-bin version:

4.0.0-1ubuntu8.15~cloud0

Hi,
I'm trying to clean up old bugs right now.
On this particular case there never was an actionable level of debug/info or a chance to recreate.
Furthermore I have not seen "similar" issues come in since then so we really would need to understand better what config/setup made this trigger the issue.

I say "similar" in quotes as the error message unfortunately is pretty generic and could mean a lot of various things. But so far all of them ended up either unreproducible
https://bugzilla.redhat.com/show_bug.cgi?id=1918211
https://bugzilla.redhat.com/show_bug.cgi?id=1821277
or misconfigurations in other layers of the virt stack which were not mentioned to be present here
https://bugzilla.redhat.com/show_bug.cgi?id=1474584

Therefore since this does not seem actionable without more insight into what triggers the issue I ahve to mark the case incomplete :-/

Mar 16 12:43:52 v001 libvirtd[14215]: 2023-03-16 11:43:52.933+0000: 14215: error : virNetSocketReadWire:1811 : End of file while reading data: Input/output error

while live migrating between two host in my 4 host cluster. Image disappeared! Not good.

What do you need from me? I have a POC Supermicro Twin2 quad node setup, with IB (FDR 56Gbps) and PCIe networking (128Gbps x8) between hosts. Migration between hosts is virtually instant over ib or dis0 (PCIe). While migrating, I triggered the above issue and the image disappeared.

How can I help debug this?

ii libvirt-bin 4.0.0-1ubuntu8.21 amd64 programs for the libvirt library

Hi Torel,
as I said above the message is pretty generic - it would be the same for a whole category of migration issues [0]. Since you seem to be able to reproduce, and do so rather quickly the next steps would be:

- for migrations you always want to trace both sides as the root cause could be on source or target
- you'd want to get detailed logs of libvirt [1]
- you'd want to get logs of qemu (let libvirt add -D to a file) [2][3]
  - guests are rather isolated by apparmor, you might need to allow it to write to that file [4]

Last but not least since you are on a pretty old libvirt (4.0 was bionic which is great but also 5 years old) you should try the very same activity with at least 22.04 Jammy and 23.04 Lunar. Chances are you have hit a fixed issue, if that is the case bisecting to what fixed it can be faster than reading massive logfiles over and over.

And since chances are quite low that you actually hit the very same issue as the initial reporter here I think filing with all that info as a new libvirt+qemu bug would be more appropriate.

[0]: https://wiki.qemu.org/Features/Migration/Troubleshooting
[1]: https://libvirt.org/kbase/debuglogs.html#turning-on-debug-logs
[2]: https://libvirt.org/formatdomain.html
[3]: https://libvirt.org/kbase/qemu-passthrough-security.html
[4]: ApparmorIsolation in https://ubuntu.com/server/docs/virtualization-libvirt