Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest

Hi Pedro,

> Wouldn't it be beneficial for all guests to be shutdown before unmounting all filesystems?

Yes on first sight that looks like a compelling suggestion.
But i've learned the hard way that usually adding dependencies eventually breaks more things than it fixes - so let us first try to understand what to expect and what happened.

The ordering at the moment includes:
- libvirt-guests.service
- virt-guest-shutdown.target
- libvirtd.service

Of these the "big one" is libvirtd.service which has
  After=network.target
  After=local-fs.target
  After=remote-fs.target
And many more which imply further lateness on startup.

libvirt-guests.service is what shuts down the guests and starts as
  After=libvirtd.service
and thereby after all of the above.

virt-guest-shutdown.target exists to allow people to hook arbitrary things into this chain in case any thing needs to be done.
Note: In most cases it is better t identify what a local system has that makes it special and set it up to go down before this. See the discussion and references from https://bugzilla.redhat.com/show_bug.cgi?id=1401054 for this.

On shutdown usually the order of these is reversed - I'd expect that libvirt-guests.service needs to stop before libvirtd.service stops (which also is reasonable as it needs to interact with libvirt).
And libvirt.service should need to be passed to leave the local/remote-FS targets on the way out, but targets might be more special in this regard.

Therefore the question becomes - which FS or FS-target was stopped/cut in your case before libvirt.service was stopped.
Could you maybe share the journal of that boot so that we can try to dissect what was going down beofore it?

You can access those with
$ journalctl --list-boots
And then select the right one and redirect it into a file like
$ journalctl -b <number> > logfile

Sure can.  I'm away but should have access to that laptop tomorrow afternoon.

Would it matter if I re-created the conditions and sent you that journal?

On Apr 21, 2021, 07:21, at 07:21, "Christian Ehrhardt " <email address hidden> wrote:
>Hi Pedro,
>
>> Wouldn't it be beneficial for all guests to be shutdown before
>unmounting all filesystems?
>
>Yes on first sight that looks like a compelling suggestion.
>But i've learned the hard way that usually adding dependencies
>eventually breaks more things than it fixes - so let us first try to
>understand what to expect and what happened.
>
>
>The ordering at the moment includes:
>- libvirt-guests.service
>- virt-guest-shutdown.target
>- libvirtd.service
>
>Of these the "big one" is libvirtd.service which has
> After=network.target
> After=local-fs.target
> After=remote-fs.target
>And many more which imply further lateness on startup.
>
>libvirt-guests.service is what shuts down the guests and starts as
> After=libvirtd.service
>and thereby after all of the above.
>
>virt-guest-shutdown.target exists to allow people to hook arbitrary
>things into this chain in case any thing needs to be done.
>Note: In most cases it is better t identify what a local system has
>that makes it special and set it up to go down before this. See the
>discussion and references from
>https://bugzilla.redhat.com/show_bug.cgi?id=1401054 for this.
>
>On shutdown usually the order of these is reversed - I'd expect that
>libvirt-guests.service needs to stop before libvirtd.service stops
>(which also is reasonable as it needs to interact with libvirt).
>And libvirt.service should need to be passed to leave the
>local/remote-FS targets on the way out, but targets might be more
>special in this regard.
>
>Therefore the question becomes - which FS or FS-target was stopped/cut
>in your case before libvirt.service was stopped.
>Could you maybe share the journal of that boot so that we can try to
>dissect what was going down beofore it?
>
>You can access those with
>$ journalctl --list-boots
>And then select the right one and redirect it into a file like
>$ journalctl -b <number> > logfile
>
>** Bug watch added: Red Hat Bugzilla #1401054
> https://bugzilla.redhat.com/show_bug.cgi?id=1401054
>
>** Changed in: libvirt (Ubuntu)
> Status: New => Incomplete
>
>--
>You received this bug notification because you are subscribed to the
>bug
>report.
>https://bugs.launchpad.net/bugs/1924494
>
>Title:
> Systemd shutdown order unmounts filesystem used by qemu/kvm guest
> before shutting down the guest
>
>Status in libvirt package in Ubuntu:
> Incomplete
>
>Bug description:
> Last night I started a reboot without manually shutting down a
> qemu/kvm guest first. After X went down a bunch of I/O errors
> associated with a network block device that contains a NTFS filesystem
> showed up on the console. I noticed that the umount.target had been
> reached but the virt-guest-shutdown.target was still waiting for the
> guest to shut down, until it timed out.
>
> Wouldn't it be beneficial for all guests to be shutdown before
> unmounting all filesystems?
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.10
> Package: sy...

Hi Christian,

Thanks for your attention. I replicated the issue and here is the
journal log for today.

Regards

Pedro Serrano

On Wed, 2021-04-21 at 11:08 +0000, Christian Ehrhardt  wrote:
> Hi Pedro,
>
> > Wouldn't it be beneficial for all guests to be shutdown before
> unmounting all filesystems?
>
> Yes on first sight that looks like a compelling suggestion.
> But i've learned the hard way that usually adding dependencies
> eventually breaks more things than it fixes - so let us first try to
> understand what to expect and what happened.
>
>
> The ordering at the moment includes:
> - libvirt-guests.service
> - virt-guest-shutdown.target
> - libvirtd.service
>
> Of these the "big one" is libvirtd.service which has
>   After=network.target
>   After=local-fs.target
>   After=remote-fs.target
> And many more which imply further lateness on startup.
>
> libvirt-guests.service is what shuts down the guests and starts as
>   After=libvirtd.service
> and thereby after all of the above.
>
> virt-guest-shutdown.target exists to allow people to hook arbitrary
> things into this chain in case any thing needs to be done.
> Note: In most cases it is better t identify what a local system has
> that makes it special and set it up to go down before this. See the
> discussion and references from
> https://bugzilla.redhat.com/show_bug.cgi?id=1401054 for this.
>
> On shutdown usually the order of these is reversed - I'd expect that
> libvirt-guests.service needs to stop before libvirtd.service stops
> (which also is reasonable as it needs to interact with libvirt).
> And libvirt.service should need to be passed to leave the
> local/remote-FS targets on the way out, but targets might be more
> special in this regard.
>
> Therefore the question becomes - which FS or FS-target was
> stopped/cut in your case before libvirt.service was stopped.
> Could you maybe share the journal of that boot so that we can try to
> dissect what was going down beofore it?
>
> You can access those with
> $ journalctl --list-boots
> And then select the right one and redirect it into a file like
> $ journalctl -b <number> > logfile
>
> ** Bug watch added: Red Hat Bugzilla #1401054
>    https://bugzilla.redhat.com/show_bug.cgi?id=1401054
>
> ** Changed in: libvirt (Ubuntu)
>        Status: New => Incomplete
>

Thanks,

In your log I see that stopping remote file systems and libvirtd properly was ordered AFTER libvirt-guests tried to shut down the guest.

I've checked this in a few ways
1. libvirt/libvirt-guests vs remote fs (e.g. NFS)
2. libvirt/libvirt-guests vs NBD

#1 Remote-FS - well ordered

Apr 23 14:37:02 ps01ubx libvirt-guests.sh[51404]: Starting shutdown on guest: psw10x64
..
Apr 23 14:39:00 ps01ubx libvirt-guests.sh[51071]: Waiting for 1 guests to shut down, 5 seconds left
Apr 23 14:39:04 ps01ubx libvirt-guests.sh[52102]: Timeout expired while shutting down domains
Then libvirtd.service stops/fails and eventually
Apr 23 14:39:05 ps01ubx systemd[1]: Stopped target Remote File Systems.

So this isn't totally looking as I first assumed which was
1. NFS with the NTFS images on it disconnected
2. shutdown guest (fails due to missing NFS)

Since the shutdown of libvirt-guests has a 120s timeout that is like a bit of breathing room in the shutdown.
A lot happens before, a lot happens after - but we see that there is nothing concurrently going on while waiting for the guest shutdown to complete.

What we see before is the I/O errors that you mentioned.
19221 Apr 23 14:37:04 ps01ubx kernel: block nbd0: NBD_DISCONNECT
19222 Apr 23 14:37:04 ps01ubx kernel: block nbd0: Disconnected due to user request.
19223 Apr 23 14:37:04 ps01ubx kernel: block nbd0: shutting down sockets
...
19229 Apr 23 14:37:04 ps01ubx kernel: blk_update_request: I/O error, dev nbd0, sector 1637875585 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0

#2 NBD - too early

Since the rest is behaving well it might be more the question "why is NTFS exiting early".
  # Set up via:
Apr 22 12:00:02 ps01ubx ntfs-3g[2059]: Mounted /dev/nbd0p2 (Read-Write, label "psData", NTFS 3.1)
  # guest shutdown starts
Apr 23 14:37:02 ps01ubx libvirt-guests.sh[51404]: Starting shutdown on guest: psw10x64
  # NBD shut down via
Apr 23 14:37:04 ps01ubx kernel: block nbd0: NBD_DISCONNECT
  # Then I/O errors happen
Apr 23 14:37:04 ps01ubx kernel: blk_update_request: I/O error, dev nbd0, sector 1637875584 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0

So it could be that NBD disconnect just waited until the guest shutdown was "kicked" but not until it completed?

The log does not contain via which config/service you have configured your NTFS/NBD.
It is just a old style /etc/fstab entry or anything more explicit in regard to ordering control like a .mount target or a custom .service?
Could you please outline in detail the config that you use for that NBD/NTFS setup?

If you have a target/unit/service for the NTFS .
If you have any kind of such could you add to your config
  Requires=virt-guest-shutdown.target
  After=virt-guest-shutdown.target
That would be what you add to anything that you want to have done/ordered into libvirt shutdown.
That should ensure correct order.

Waiting to see how your NTFS setup looks like ...

For your experiments, here a few examples
1. using a mount unit for ntfs
   https://brunofontes.net/enu/post/mounting-partitions-systemd
2. doing disk setup with a service
   https://unix.stackexchange.com/questions/608431/systemd-fix-ordering-cycle-when-mounting-an-ntfs-volume-on-a-legacy-raid-array

General help on mount units
  https://www.freedesktop.org/software/systemd/man/systemd.mount.html

And as I said, for this case once you have such unites you can control them fine grained via:

a) be completely shut down before libvirt can end (not what you want actually as that can be running concurrent to libvirt-guest service - this is better for other things to do before guests shut down - like swap scrubbing for secrecy or such)
  Requires=virt-guest-shutdown.target
  After=virt-guest-shutdown.target

b) Ensure to be started before libvirt and even more important, only end after them on shutdown (reverse order)
  Before=libvirtd.service libvirt-guests.service

Christian

I set this up back when without properly researching systemd.mount, particularly the automatic dependencies. I make half of the changes you recommended (Requires=virt-guest-shutdown.target; After=virt-guest-shutdown.target) and it made no change, the I/O errors still showed up. I'm attaching the custom systemd services and the libvirt hook for your information.

But, instead of following in the direction of a custom setup I will transition this to /etc/fstab parent/child mounts (the child mount is a .vhd file on the parent fs that will now be mounted using the loop device instead of nbd) and a simpler libvirt hook with mount/umount functionality. Sorry for making this more complicated than they should have been.

Regards

Pedro

And by the way, I just tested the old setup with your full recommendation, adding the BEFORE= statement and it finally stopped the I/O errors. I can now take the time to transition properly.

Thank you!

Pedro

> And by the way, I just tested the old setup with your full
> recommendation, adding the BEFORE= statement and it finally stopped the
> I/O errors. I can now take the time to transition properly.

Awesome, glad I could help!
And it might help others who find this bug if affected by a similar case.