Some modprobe loading services requested by the pstore service fail

Staged for kinetic: https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?h=ubuntu-kinetic&id=46c36d4c73df8980f6b6137142fb16ba90465a94

Alfonso, would this also affect Focal (Ubuntu Core 20), after this patch is shipped to that series via SRU (as it happened on Jammy already)?

https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?h=ubuntu-focal&id=6e60756f2079d6408abdb967127a1d9b9a0eba8c

Do we need to include this fix for Focal too, before uploading the next Focal SRU?

On Mon, Jul 25, 2022 at 4:00 PM Lukas Märdian <email address hidden>
wrote:

> Alfonso, would this also affect Focal (Ubuntu Core 20), after this patch
> is shipped to that series via SRU (as it happened on Jammy already)?
>
> https://git.launchpad.net/~ubuntu-core-
> dev/ubuntu/+source/systemd/commit/?h=ubuntu-
> focal&id=6e60756f2079d6408abdb967127a1d9b9a0eba8c
>
> Do we need to include this fix for Focal too, before uploading the next
> Focal SRU?
>

Although the bug was reported for UC22, if the patch causing the problem is
in focal I would say that yes, probably it would be needed to backport it
there.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1982462
>
> Title:
> Some modprobe loading services requested by the pstore service fail
>
> Status in systemd package in Ubuntu:
> In Progress
> Status in systemd source package in Jammy:
> Triaged
> Status in systemd source package in Kinetic:
> In Progress
>
> Bug description:
> [Impact]
>
> It has been detected that some modprobe services fail on UC22 after
> the jammy upgrade 249.11-0ubuntu3.4:
>
> $ systemctl --system --no-ask-password --no-pager list-units
> --state=failed
> Failed units:
> UNIT LOAD ACTIVE SUB DESCRIPTION
> ● modprobe@chromeos_pstore.service loaded failed failed Load Kernel
> Module chromeos_pstore
> ● modprobe@efi_pstore.service loaded failed failed Load Kernel
> Module efi_pstore
> ● <email address hidden> loaded failed failed Load Kernel
> Module mtdpstore
> ● modprobe@pstore_blk.service loaded failed failed Load Kernel
> Module pstore_blk
> ● modprobe@pstore_zone.service loaded failed failed Load Kernel
> Module pstore_zone
> ● <email address hidden> loaded failed failed Load Kernel
> Module ramoops
>
> This happens because of some changes to systemd-pstore.service that
> now has:
>
> After=modprobe@efi_pstore.service <email address hidden>
> modprobe@chromeos_pstore.service <email address hidden>
> modprobe@pstore_zone.service modprobe@pstore_blk.service
> Wants=modprobe@efi_pstore.service <email address hidden>
> modprobe@chromeos_pstore.service <email address hidden>
> modprobe@pstore_zone.service modprobe@pstore_blk.service
>
> This causes too many tries of the modprobe services, that fail in the
> end with
>
> Jul 20 09:02:39 ubuntu systemd[1]: modprobe@chromeos_pstore.service:
> Start request repeated too quickly.
>
> Although we have seen this only on UC22, it potentially can affect
> classic systems as well, as systemd-pstore.service is re-tried there a
> few times too. See https://github.com/snapcore/core-base/issues/72 for
> more details.
>
> A fix for this is available upstream:
>
> https://github.com/systemd/systemd/commit/9625350e5381a68c1179ae4581e7586c206663e1
>
> [Test Plan]
>
> Start the device and check that there is no modprobe-pstore related
> failed service. This is racy, so a few tries will be needed to make
> sure things are fine.
>
> [Where problems could occur]
>
> The modprobe services are usually depende...

This bug was fixed in the package systemd - 251.2-2ubuntu2

---------------
systemd (251.2-2ubuntu2) kinetic; urgency=medium

  [ Lukas Märdian ]
  * Remove restart limit on the modprobe@.service (LP: #1982462)
    File: debian/patches/units-remove-the-restart-limit-on-the-modprobe-.service.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=46c36d4c73df8980f6b6137142fb16ba90465a94

  [ Nick Rosbrook ]
  * core,firstboot: workaround timezone issues on Ubuntu Core (LP: #1981042)
    Thanks to Robert Ancell for preparing the patch.
    File: debian/patches/lp1981042-core-firstboot-workaround-timezone-issues-caused-by-Ubunt.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=9a435d43e753f39531c9a5517a85e8eb259e18f1
  * pstore: do not try to load mtdpstore (LP: #1981622)
    File: debian/patches/lp1978079-pstore-Run-after-modules-are-loaded.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=38d8d12be924477f0eecb64f3737e07b03d73a42

 -- Nick Rosbrook <email address hidden> Fri, 22 Jul 2022 15:23:45 -0400

I have tested a modified systemd-pstore.service in UC20 (focal based) adding:

[Unit]
After=modprobe@efi_pstore.service <email address hidden> modprobe@chromeos_pstore.service <email address hidden> modprobe@pstore_zone.service modprobe@pstore_blk.service
Wants=modprobe@efi_pstore.service <email address hidden> modprobe@chromeos_pstore.service <email address hidden> modprobe@pstore_zone.service modprobe@pstore_blk.service

As a result I get failures in the services trying to load modules for pstore, for instance:

ubuntu@ubuntu:~$ sudo systemctl status <email address hidden>
● <email address hidden> - Load Kernel Module mtdpstore
     Loaded: loaded (/lib/systemd/system/modprobe@.service; static; vendor preset: enabled)
     Active: inactive (dead) since Thu 2022-08-04 15:59:31 UTC; 25s ago
       Docs: man:modprobe(8)
    Process: 400 ExecStart=/sbin/modprobe -abq mtdpstore (code=exited, status=1/FAILURE)
   Main PID: 400 (code=exited, status=1/FAILURE)

Aug 04 15:59:31 ubuntu systemd[1]: <email address hidden>: Succeeded.
Aug 04 15:59:31 ubuntu systemd[1]: Finished Load Kernel Module mtdpstore.

Which is fine, but there is the potential of too many successive restart of these services, which would result in https://github.com/systemd/systemd/issues/23742. To prevent this, https://github.com/systemd/systemd/commit/9625350e5381a68c1179ae4581e7586c206663e1 should be included in the next systemd focal SRU.

Status changed to 'Confirmed' because the bug affects multiple users.

Hello Alfonso, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Alfonso, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I have tested core22 built using the proposed pocket and I can confirm that the issue is gone (no failed services on start up anymore).

I have also tested the jammy deb on classic and the modprobe services do not fail either:

root@jammy:~# apt policy systemd
systemd:
  Installed: 249.11-0ubuntu3.5
  Candidate: 249.11-0ubuntu3.5
  Version table:
 *** 249.11-0ubuntu3.5 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.4 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

I have tested core20 built using the proposed pocket and I can confirm that the issue does not appear (no failed services on start up).

I have also tested the focal deb on classic and the modprobe services do not fail either:

root@focal:~# apt policy systemd
systemd:
  Installed: 245.4-4ubuntu3.18
  Candidate: 245.4-4ubuntu3.18
  Version table:
 *** 245.4-4ubuntu3.18 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     245.4-4ubuntu3.17 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
     245.4-4ubuntu3.15 500
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     245.4-4ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.5) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.48.2-0ubuntu1 (ppc64el)
systemd/249.11-0ubuntu3.5 (ppc64el)
udisks2/2.9.4-1ubuntu2 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.18) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

linux-gke-5.15/5.15.0-1015.18~20.04.1 (arm64, amd64)
netplan.io/0.104-0ubuntu2~20.04.2 (armhf)
linux-hwe-5.15/5.15.0-48.54~20.04.1 (armhf, amd64)
gvfs/1.44.1-1ubuntu1.1 (ppc64el)
linux-oracle-5.15/5.15.0-1016.20~20.04.1 (arm64)
linux-oracle-5.13/5.13.0-1036.43~20.04.1 (arm64)
linux-hwe-5.11/5.11.0-61.61 (arm64)
systemd/245.4-4ubuntu3.18 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Hello Alfonso, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.6) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

stunnel4/3:5.63-1build1 (amd64)
munin/2.0.57-1ubuntu2 (amd64)
corosync/unknown (s390x)
conntrack-tools/unknown (s390x)
exim4/4.95-4ubuntu2.1 (ppc64el)
umockdev/0.17.7-1 (armhf)
netplan.io/0.104-0ubuntu2.1 (amd64)
initramfs-tools/0.140ubuntu13 (amd64)
dovecot/unknown (s390x)
network-manager/1.36.6-0ubuntu2 (amd64)
cups/unknown (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Since the 249.11-0ubuntu3.6 upload of systemd only contains a change to skip an autopkgtest, Alfonso's previous verification on jammy using 249.11-0ubuntu3.6 is still valid (https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1982462/comments/11).

The autopkgtest regressions for systemd 249.11-0ubuntu3.6 in jammy-proposed were all resolved with retries.

This bug was fixed in the package systemd - 249.11-0ubuntu3.6

---------------
systemd (249.11-0ubuntu3.6) jammy; urgency=medium

  * Deny-list TEST-58-REPART on ppc64el (LP: #1988994)
    File: debian/patches/lp1988994-Deny-list-TEST-58-REPART-on-ppc64el.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d2ed3cc1d223bf35015b15ff83b50156b58f0f38

systemd (249.11-0ubuntu3.5) jammy; urgency=medium

  [ Nick Rosbrook ]
  * Ensure dns_search_domain_unlink_marked removes all marked domains (LP: #1975667)
    File: debian/patches/lp1975667-Ensure-dns_search_domain_unlink_marked-removes-all-marked.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=919d5ddedd5bb8b45ab9437bf42d66c2821bb074
  * core,firstboot: workaround timezone issues on Ubuntu Core (LP: #1981042)
    Thanks to Robert Ancell for preparing the patch.
    File: debian/patches/lp1981042-core-firstboot-workaround-timezone-issues-caused-by-Ubunt.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b15546361b549217908fb6ca5d473be23d7fa757
  * network: do not remove localhost address (LP: #1979951)
    File: debian/patches/lp1979951-network-do-not-remove-localhost-address.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=2cd88391cce9fe95a486ae6dd214c12f236f3881
  * units: remove the restart limit on the modprobe@.service (LP: #1982462)
    File: debian/patches/lp1982462-units-remove-the-restart-limit-on-the-modprobe-.service.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8f0acd1b2fbb8eed1259c34963e5e9b201bef900
  * pstore: do not try to load mtdpstore (LP: #1981622)
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=15225032c3657f5906ee49d48929f9295a8664a0
  * core/mount: downgrade log level about several mkdir failures (LP: #1979952)
    Files:
    - debian/patches/lp1979952-Revert-core-mount-fail-early-if-directory-cannot-be-creat.patch
    - debian/patches/lp1979952-core-mount-downgrade-log-level-about-several-mkdir-failur.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ee8cfcf500698fab2e990de291ecf4c3ab87a4ae
  * debian/control: add Recommends: systemd-hwe-hwdb to udev.
    The systemd-hwe-hwdb brings in additional hwdb rules for HWE, so we want
    those installed with udev by default.
    File: debian/control
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4a7a3258c33201cca305956820fcc6bcd6052d76
  * hwdb: implement --root option for systemd-hwdb query (LP: #1988078)
    Files:
    - debian/libsystemd0.symbols
    - debian/patches/lp1988078-hwdb-implement-root-option-for-systemd-hwdb-query.patch
    - debian/patches/lp1988078-sd-hwdb-add-sd_hwdb_new_from_path.patch
    - debian/patches/lp1988078-sd-hwdb-include-sys-stat.h-in-hwdb-internal.h.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=937fef96c858f2f2042bf71032f315647c14add0

  [ Luca Boccassi ]
  * Enable systemd-repart and ship it in a new s...

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package systemd - 245.4-4ubuntu3.18

---------------
systemd (245.4-4ubuntu3.18) focal; urgency=medium

  [ Nick Rosbrook ]
  * core: make sure we don't get confused when setting TERM for a tty fd
    (LP: #1959475)
    File: debian/patches/lp1959475-core-make-sure-we-don-t-get-confused-when-setting-TERM-fo.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b10c6853050dde26665caf3b15444d768d2bc498
  * shared/calendarspec: when mktime() moves us backwards, jump forward
    (LP: #1966800)
    File: debian/patches/lp1966800-shared-calendarspec-when-mktime-moves-us-backwards-jump-f.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1f063541e44f6ff1a6904676d4264a2e49a09594
  * network: do not remove localhost address (LP: #1979951)
    File: debian/patches/lp1979951-network-do-not-remove-localhost-address.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=87f872b8c5451f353601fb606e7fd7a479217cef
  * units: remove the restart limit on the modprobe@.service (LP: #1982462)
    File: debian/patches/lp1982462-units-remove-the-restart-limit-on-the-modprobe-.service.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=68353ffaf3539e6a58ef62a8b50850f56eae29ea

  [ Mustafa Kemal Gilor ]
  * d/p/lp1978079-efi-pstore-not-cleared-on-boot.patch: pstore: Run after
    modules are loaded. Thanks to Alexander Graf <email address hidden>.
    (LP: #1978079)
    Author: Mustafa Kemal Gilor
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6e60756f2079d6408abdb967127a1d9b9a0eba8c

 -- Nick Rosbrook <email address hidden> Wed, 31 Aug 2022 11:27:33 -0400