ephemeral-disk-warning.service causes ordering cycle on multi-user.target

Bug #1624596 reported by Scott Moser
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
walinuxagent (Ubuntu)
Fix Released
Medium
Unassigned
Xenial
Fix Released
Medium
Scott Moser

Bug Description

=== Begin SRU Template ===
[Impact]
Some recent systemd related cloud-init changes in 0.7.7-28-g34a26f7-0ubuntu1
and later caused systemd 'ordering cycle' around the walinuxagent provided
ephemeral-disk-warning.service .

The result would be that journalctl would report:
   Breaking ordering cycle by deleting job ephemeral-disk-warning.service/start
Due to a order cycle

 * An explanation of the effects of the bug on users and

 * justification for backporting the fix to the stable release.

 * In addition, it is helpful, but not required, to include an
   explanation of how the upload fixes this bug.

[Test Case]
Boot system on azure, then run:

$ journalctl | grep -i "ordering cycle"

There *should* be no ordering cycles in our boot.

$ systemctl status ephemeral-disk-warning.service | grep Condition
Condition: start condition failed at Tue 2016-09-20 16:15:03 UTC; 37min ago

That is the failure, to show it functional and fixed

$ echo "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main" |
   sudo tee /etc/apt/sources.list.d/proposed.list
$ sudo apt-get update -qy
$ sudo apt-get -qy install walinuxagent
$ dpkg-query --show walinuxagent
$ sudo reboot

Then repeat the above, and you should see no 'ordering cycle' messages, and

$ systemctl status ephemeral-disk-warning.service | grep Active
   Active: active (exited) since Tue 2016-09-20 17:08:46 UTC; 1h 0min ago

And the file that gets created should be there.
$ ls -l /mnt/DATALOSS_WARNING_README.txt
-r--r--r-- 1 root root 639 Sep 20 17:08 /mnt/DATALOSS_WARNING_README.txt

[Regression Potential]
Changing order of boot is quite often painful due to subtle changes and
the overall newness of systemd. The changes here are fairly well tested
and fix an issue. Worst case, that job does not run (as it does not run now)
and we do not get the file created.

[Other Info]
This is fallout of fix for bug 1576692.
Full description of the fix is:
    debian/ephemeral-disk-warning.service: avoid systemd ordering cycle

    ephemeral-disk-warning.service had 'WantedBy' of multi-user.target
    which implies "Before" of multi-user.target. It also had
    After of cloud-final.service. Since cloud-final.service is now
    After multi-user.target, that caused the cycle.

    Relax this to be only After cloud-config.service which will run
    before multi-user.target does and avoid the cycle.

    Also add a 'Description'.

    LP: #1624596
--- End SRU Template ---

$ journalctl | grep break -a2 -b8
42956-Sep 17 01:45:02 smoser0917x systemd[1]: Detected virtualization microsoft.
43031-Sep 17 01:45:02 smoser0917x systemd[1]: Detected architecture x86-64.
43101-Sep 17 01:45:02 smoser0917x systemd[1]: Set hostname to <smoser0917x>.
43172-Sep 17 01:45:02 smoser0917x systemd[1]: multi-user.target: Found ordering cycle on multi-user.target/start
43279-Sep 17 01:45:02 smoser0917x systemd[1]: multi-user.target: Found dependency on ephemeral-disk-warning.service/start
43395-Sep 17 01:45:02 smoser0917x systemd[1]: multi-user.target: Found dependency on cloud-final.service/start
43500-Sep 17 01:45:02 smoser0917x systemd[1]: multi-user.target: Found dependency on multi-user.target/start
43603-Sep 17 01:45:02 smoser0917x systemd[1]: multi-user.target: Breaking ordering cycle by deleting job ephemeral-disk-warning.service/start
43739:Sep 17 01:45:02 smoser0917x systemd[1]: ephemeral-disk-warning.service: Job ephemeral-disk-warning.service/start deleted to break ordering cycle starting with multi-user.target/start
43922-Sep 17 01:45:02 smoser0917x systemd[1]: Listening on udev Kernel Socket.
43995-Sep 17 01:45:02 smoser0917x systemd[1]: Reached target Encrypted Volumes.
44069-Sep 17 01:45:02 smoser0917x systemd[1]: Listening on Syslog Socket.
44137-Sep 17 01:45:02 smoser0917x systemd[1]: Started Forward Password Requests to Wall Directory Watch.
44236-Sep 17 01:45:02 smoser0917x systemd[1]: Listening on Journal Socket.
44305-Sep 17 01:45:02 smoser0917x systemd[1]: Listening on LVM2 poll daemon socket.
44383-Sep 17 01:45:02 smoser0917x systemd[1]: Reached target Swap.
44444-Sep 17 01:45:02 smoser0917x systemd[1]: Listening on fsck to fsckd communication Socket.

$ dpkg-query --show cloud-init walinuxagent
cloud-init 0.7.8-1-g3705bb5-0ubuntu1~16.04.1
walinuxagent 2.1.3-0ubuntu4.1

Related bugs:
 * bug 1576692: fully support package installation in systemd

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: walinuxagent 2.1.3-0ubuntu4.1
ProcVersionSignature: User Name 4.4.0-36.55-generic 4.4.16
Uname: Linux 4.4.0-36-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
Date: Sat Sep 17 01:51:22 2016
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: walinuxagent
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :

Seems to fix:
$ systemctl status ephemeral-disk-warning.service
● ephemeral-disk-warning.service
   Loaded: loaded (/lib/systemd/system/ephemeral-disk-warning.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sat 2016-09-17 01:57:46 UTC; 3min 29s ago
  Process: 2929 ExecStart=/usr/sbin/ephemeral-disk-warning.sh (code=exited, status=0/SUCCESS)
 Main PID: 2929 (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/ephemeral-disk-warning.service

Sep 17 01:57:46 smoser0917x systemd[1]: Starting ephemeral-disk-warning.service...
Sep 17 01:57:46 smoser0917x systemd[1]: Started ephemeral-disk-warning.service.

$ diff -u /lib/systemd/system/ephemeral-disk-warning.service.dist /lib/systemd/system/ephemeral-disk-warning.service
--- /lib/systemd/system/ephemeral-disk-warning.service.dist 2016-09-17 02:02:16.005185102 +0000
+++ /lib/systemd/system/ephemeral-disk-warning.service 2016-09-17 02:02:08.997530604 +0000
@@ -1,6 +1,6 @@
 [Unit]
 Description=
-After=cloud-final.service
+After=cloud-config.service
 ConditionVirtualization=microsoft
 ConditionPathIsMountPoint=/mnt
 ConditionPathExists=/dev/disk/azure/resource-part1

Changed in walinuxagent (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
Scott Moser (smoser)
Changed in walinuxagent (Ubuntu Xenial):
assignee: nobody → Scott Moser (smoser)
importance: Undecided → Medium
status: New → Confirmed
Scott Moser (smoser)
description: updated
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package walinuxagent - 2.1.5-0ubuntu3

---------------
walinuxagent (2.1.5-0ubuntu3) yakkety; urgency=medium

  * debian/ephemeral-disk-warning.service: add a description
  * debian/ephemeral-disk-warning.service: fix ordering cycle
    (LP: #1624596)
  * d/66-azure-storage.rules, d/install: fix dangling symlink in
    at /lib/udev/rules.d/66-azure-storage.rules.

 -- Scott Moser <email address hidden> Tue, 20 Sep 2016 13:50:44 -0400

Changed in walinuxagent (Ubuntu):
status: Triaged → Fix Released
Scott Moser (smoser)
description: updated
Revision history for this message
Scott Moser (smoser) wrote :
Scott Moser (smoser)
description: updated
Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Hello Scott, or anyone else affected,

Accepted walinuxagent into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/walinuxagent/2.1.3-0ubuntu4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in walinuxagent (Ubuntu Xenial):
status: Confirmed → Fix Committed
Revision history for this message
Scott Moser (smoser) wrote :

$ cat /etc/cloud/build.info
build_name: server
serial: 20160927

$ dpkg-query --show cloud-init walinuxagent
cloud-init 0.7.8-1-g3705bb5-0ubuntu1~16.04.1
walinuxagent 2.1.3-0ubuntu4.1

$ ls /mnt/
lost+found

$ journalctl | grep -i "ordering cycle"
Sep 28 18:56:58 ubuntu systemd[1]: multi-user.target: Found ordering cycle on multi-user.target/start
Sep 28 18:56:58 ubuntu systemd[1]: multi-user.target: Breaking ordering cycle by deleting job ephemeral-disk-warning.service/start
Sep 28 18:56:58 ubuntu systemd[1]: ephemeral-disk-warning.service: Job ephemeral-disk-warning.service/start deleted to break ordering cycle starting with multi-user.target/start

$ echo "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main" | sudo tee /etc/apt/sources.list.d/proposed.list
deb http://archive.ubuntu.com/ubuntu xenial-proposed main

$ sudo apt-get update -qy
$ sudo apt-get install -qy walinuxagent

$ dpkg-query --show walinuxagent
walinuxagent 2.1.3-0ubuntu4.2

$ sudo reboot
...

$ ls -l /mnt/
total 20
-r--r--r-- 1 root root 639 Sep 28 19:04 DATALOSS_WARNING_README.txt
drwx------ 2 root root 16384 Sep 28 18:57 lost+found

$ journalctl | grep -i "ordering cycle" || echo "no cycles"
no cycles

$ systemctl status ephemeral-disk-warning.service | grep Active
   Active: active (exited) since Wed 2016-09-28 19:07:22 UTC; 30s ago

tags: added: verification-done
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package walinuxagent - 2.1.3-0ubuntu4.2

---------------
walinuxagent (2.1.3-0ubuntu4.2) xenial-proposed; urgency=medium

  * debian/ephemeral-disk-warning.service: add a description
  * debian/ephemeral-disk-warning.service: fix ordering cycle
    (LP: #1624596)

 -- Scott Moser <email address hidden> Wed, 21 Sep 2016 14:11:50 -0400

Changed in walinuxagent (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for walinuxagent has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.