Bug #1931104 “Test of dogtag-pki is failing on s390x due to LTO” : Bugs : nss package : Ubuntu

Christian Ehrhardt  (paelzer) on 2021-06-07

description:

updated

Christian Ehrhardt  (paelzer) on 2021-06-07

tags:

added: update-excuse

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-06-07:

#1

FYI by tjaaltonen - there's another crasher in 3.66 on ppc64el.. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989410

So 3.66 won't be the "take this and it works" solution.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-06-07:

#2

While we wait for 3.67 and maybe (Thanks Timo) for [1] I have ensured that we have a 3.66 test build.
=> https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4577/+packages

The check of it's delta also showed that we can drop a bit of it nowadays.
=> https://code.launchpad.net/~paelzer/ubuntu/+source/nss/+git/nss/+ref/merge-impish-3.66-1

[1]: https://phabricator.services.mozilla.com/D116274

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-06-08:

#3

I was able to verify that a merge of 3.66 would on Ubuntu trigger the very same bug that Debian has blocking the dogtag-pki test on powerpc64.

=> https://autopkgtest.ubuntu.com/results/autopkgtest-impish-ci-train-ppa-service-4577/impish/ppc64el/d/dogtag-pki/20210608_031158_a9d4a@/log.gz

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-06-08:

#4

There is another fix in master that belongs to https://bugzilla.mozilla.org/show_bug.cgi?id=1566124 - I've bumped my PPA build to include both as it is worth a try if this fixes the current ppc64 issues in v3.66.

Build of 3.66-1ubuntu1~impishppa2 started, later on I'll let the autopkgtests run.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-06-09:

#5

This feels like a circle with nss/2:3.66-1ubuntu1~impishppa2 in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4577/+packages now ppc64 works but s390x fails with (on the surface) the same symptom as it started with in 3.63 :-/

I retriggered the tests to see if that is flaky or reproducible.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-06-10:

#6

Reproducible, ppc64 is fixed and s390x broken by the latest 3.66 + fixes from master.
Links:
https://autopkgtest.ubuntu.com/results/autopkgtest-impish-ci-train-ppa-service-4577/impish/s390x/d/dogtag-pki/20210609_065306_2e698@/log.gz
https://autopkgtest.ubuntu.com/results/autopkgtest-impish-ci-train-ppa-service-4577/impish/s390x/d/dogtag-pki/20210609_121610_a71da@/log.gz
https://autopkgtest.ubuntu.com/results/autopkgtest-impish-ci-train-ppa-service-4577/impish/s390x/d/dogtag-pki/20210609_124415_d071b@/log.gz

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-06-10:

#7

Without the recent PPC fixes s390x was broken as well, so I'm not saying "the ppc fixes broke s390x":
https://autopkgtest.ubuntu.com/results/autopkgtest-impish-ci-train-ppa-service-4577/impish/s390x/d/dogtag-pki/20210608_073451_f187d@/log.gz

Instead there seems to be a new crash affecting s390x that has to be looked at to be able to rev up the nss version.

Tagging server-next and subscribing the team as this is no more a +1 task.
If we are lucky the always dilligent locutus (who often merges nss) might come by and give it a short as well when we have latter nss versions.

tags:

added: server-next

Paride Legovini (paride) on 2021-06-16

Changed in nss (Ubuntu):
assignee:	nobody → Paride Legovini (paride)

Revision history for this message

Paride Legovini (paride) wrote on 2021-07-21:

#8

After some work and testing, here are my findings.

- The dogtag-pki autopkgtest failure is manually reproducible
   using autopkgtest-virt-lxd.
- The dogtag-pki autopkgtests pass with in Impish using
   libnss3 *from the archive* (uploaded and built on Hirsute).
- The dogtag-pki autopkgtests FAIL when using the very same
   libnss3 version but rebuilt from source on a Hirsute schroot.
   The debdiff between the two binary packages is:

File lists identical (after any substitutions)

Control files: lines which differ (wdiff format)
------------------------------------------------
Installed-Size: [-4255-] {+4318+}

- This seems to be s390x-specific (I can't reproduce on my
   amd64 laptop)
- I tried merging the latest version of src:nss from Debian,
   which required refreshing a s390x-specific patch, so I was
   really hoping this would fix it, but no: it fails in the
   same way.

FWIW I have a branch ready for merging 2:3.67-2 to Impish, but
I'm not sure on how to handle this dogtag-pki autopkgtest failure.

Revision history for this message

Paride Legovini (paride) wrote on 2021-07-21:

#9

Rebuilding the same source package on Groovy produces a "good" libnss3 package (the Impish dogtag-pki autopkgtests pass when using it). This means that a build-dependency that was upgraded in Hirsute caused the regression.

(As Christian made me notice rebuilding on Hirsute *release* with no updates may still produce a broken package, even if the "good" package in the archive was built on Hirsute. This is because back when the Hirsute package was built Hirsute was still in development. The true way to go back to what Hirsute was initially is to go back to Groovy.)

Revision history for this message

Paride Legovini (paride) wrote on 2021-07-21:

#10

Download full text (8.3 KiB)

Diff of the sbuild Installed-Build-Depends from the "good" Hirsute build that produced the nss packages now in the archive and a "bad" Hirsute build done in an up-to-date Hirsute schroot:

--- good 2021-07-21 12:02:03.870339411 +0200
+++ bad 2021-07-21 12:03:20.367850047 +0200
@@ -3,38 +3,39 @@
  automake (= 1:1.16.3-2ubuntu1),
  autopoint (= 0.21-3ubuntu2),
  autotools-dev (= 20180224.1+nmu1),
- base-files (= 11ubuntu16),
+ base-files (= 11ubuntu19),
  base-passwd (= 3.5.49),
- bash (= 5.1-1ubuntu1),
- binutils (= 2.36.1-0ubuntu1),
- binutils-common (= 2.36.1-0ubuntu1),
- binutils-s390x-linux-gnu (= 2.36.1-0ubuntu1),
- bsdextrautils (= 2.36.1-1ubuntu2),
- bsdutils (= 1:2.36.1-1ubuntu2),
+ bash (= 5.1-2ubuntu1),
+ binutils (= 2.36.1-6ubuntu1),
+ binutils-common (= 2.36.1-6ubuntu1),
+ binutils-s390x-linux-gnu (= 2.36.1-6ubuntu1),
+ bsdextrautils (= 2.36.1-7ubuntu2),
+ bsdutils (= 1:2.36.1-7ubuntu2),
  build-essential (= 12.8ubuntu3),
- bzip2 (= 1.0.8-4ubuntu2),
+ bzip2 (= 1.0.8-4ubuntu3),
  coreutils (= 8.32-4ubuntu2),
- cpp (= 4:10.2.0-1ubuntu1),
- cpp-10 (= 10.2.1-19ubuntu1),
+ cpp (= 4:10.3.0-1ubuntu1),
+ cpp-10 (= 10.3.0-1ubuntu1),
  dash (= 0.5.11+git20200708+dd9ef66+really0.5.11+git20200708+dd9ef66-5ubuntu1),
  debconf (= 1.5.74),
- debhelper (= 13.3.3ubuntu2),
+ debhelper (= 13.3.4ubuntu1),
  debianutils (= 4.11.2),
+ debugedit (= 1:0.1-0ubuntu2),
  dh-autoreconf (= 20),
- dh-exec (= 0.23.2),
+ dh-exec (= 0.23.4),
  dh-strip-nondeterminism (= 1.11.0-1),
  diffutils (= 1:3.7-3ubuntu1),
- dpkg (= 1.20.7.1ubuntu2),
- dpkg-dev (= 1.20.7.1ubuntu2),
- dwz (= 0.13+20210201-1),
+ dpkg (= 1.20.9ubuntu1),
+ dpkg-dev (= 1.20.9ubuntu1),
+ dwz (= 0.14-1),
  file (= 1:5.39-3),
- findutils (= 4.7.0-1ubuntu2),
- g++ (= 4:10.2.0-1ubuntu1),
- g++-10 (= 10.2.1-19ubuntu1),
- gcc (= 4:10.2.0-1ubuntu1),
- gcc-10 (= 10.2.1-19ubuntu1),
- gcc-10-base (= 10.2.1-19ubuntu1),
- gcc-11-base (= 11-20210207-1ubuntu1),
+ findutils (= 4.8.0-1ubuntu1),
+ g++ (= 4:10.3.0-1ubuntu1),
+ g++-10 (= 10.3.0-1ubuntu1),
+ gcc (= 4:10.3.0-1ubuntu1),
+ gcc-10 (= 10.3.0-1ubuntu1),
+ gcc-10-base (= 10.3.0-1ubuntu1),
+ gcc-11-base (= 11.1.0-1ubuntu1~21.04),
  gettext (= 0.21-3ubuntu2),
  gettext-base (= 0.21-3ubuntu2),
  grep (= 3.6-1),
@@ -43,113 +44,114 @@
  hostname (= 3.23),
  init-system-helpers (= 1.60),
  intltool-debian (= 0.35.0+20060710.5),
- libacl1 (= 2.2.53-10),
+ libacl1 (= 2.2.53-10ubuntu1),
  libarchive-zip-perl (= 1.68-1),
- libasan6 (= 10.2.1-19ubuntu1),
- libatomic1 (= 11-20210207-1ubuntu1),
- libattr1 (= 1:2.4.48-6),
+ libasan6 (= 11.1.0-1ubuntu1~21.04),
+ libatomic1 (= 11.1.0-1ubuntu1~21.04),
+ libattr1 (= 1:2.4.48-6build1),
  libaudit-common (= 1:3.0-2ubuntu1),
  libaudit1 (= 1:3.0-2ubuntu1),
- libbinutils (= 2.36.1-0ubuntu1),
- libblkid1 (= 2.36.1-1ubuntu2),
- libbz2-1.0 (= 1.0.8-4ubuntu2),
- libc-bin (= 2.33-0ubuntu2),
- libc-dev-bin (= 2.33-0ubuntu2),
- libc6 (= 2.33-0ubuntu2),
- libc6-dev (= 2.33-0ubuntu2),
+ libbinutils (= 2.36.1-6ubuntu1),
+ libblkid1 (= 2.36.1-7ubuntu2),
+ libbz2-1.0 (= 1.0.8-4ubuntu3),
+ libc-bin (= 2.33-0ubuntu5),
+ libc-dev-bin (= 2.33-0ubuntu5),
+ libc6 (= 2.33-0ubuntu5),
+ libc6-dev (= 2.33-0ubuntu5),
  libcap-ng0 (= 0.7.9-2.2build1),
...

Diff of the sbuild Installed-Build-Depends from the "good" Hirsute build that produced the nss packages now in the archive and a "bad" Hirsute build done in an up-to-date Hirsute schroot:

--- good	2021-07-21 12:02:03.870339411 +0200
+++ bad	2021-07-21 12:03:20.367850047 +0200
@@ -3,38 +3,39 @@
  automake (= 1:1.16.3-2ubuntu1),
  autopoint (= 0.21-3ubuntu2),
  autotools-dev (= 20180224.1+nmu1),
- base-files (= 11ubuntu16),
+ base-files (= 11ubuntu19),
  base-passwd (= 3.5.49),
- bash (= 5.1-1ubuntu1),
- binutils (= 2.36.1-0ubuntu1),
- binutils-common (= 2.36.1-0ubuntu1),
- binutils-s390x-linux-gnu (= 2.36.1-0ubuntu1),
- bsdextrautils (= 2.36.1-1ubuntu2),
- bsdutils (= 1:2.36.1-1ubuntu2),
+ bash (= 5.1-2ubuntu1),
+ binutils (= 2.36.1-6ubuntu1),
+ binutils-common (= 2.36.1-6ubuntu1),
+ binutils-s390x-linux-gnu (= 2.36.1-6ubuntu1),
+ bsdextrautils (= 2.36.1-7ubuntu2),
+ bsdutils (= 1:2.36.1-7ubuntu2),
  build-essential (= 12.8ubuntu3),
- bzip2 (= 1.0.8-4ubuntu2),
+ bzip2 (= 1.0.8-4ubuntu3),
  coreutils (= 8.32-4ubuntu2),
- cpp (= 4:10.2.0-1ubuntu1),
- cpp-10 (= 10.2.1-19ubuntu1),
+ cpp (= 4:10.3.0-1ubuntu1),
+ cpp-10 (= 10.3.0-1ubuntu1),
  dash (= 0.5.11+git20200708+dd9ef66+really0.5.11+git20200708+dd9ef66-5ubuntu1),
  debconf (= 1.5.74),
- debhelper (= 13.3.3ubuntu2),
+ debhelper (= 13.3.4ubuntu1),
  debianutils (= 4.11.2),
+ debugedit (= 1:0.1-0ubuntu2),
  dh-autoreconf (= 20),
- dh-exec (= 0.23.2),
+ dh-exec (= 0.23.4),
  dh-strip-nondeterminism (= 1.11.0-1),
  diffutils (= 1:3.7-3ubuntu1),
- dpkg (= 1.20.7.1ubuntu2),
- dpkg-dev (= 1.20.7.1ubuntu2),
- dwz (= 0.13+20210201-1),
+ dpkg (= 1.20.9ubuntu1),
+ dpkg-dev (= 1.20.9ubuntu1),
+ dwz (= 0.14-1),
  file (= 1:5.39-3),
- findutils (= 4.7.0-1ubuntu2),
- g++ (= 4:10.2.0-1ubuntu1),
- g++-10 (= 10.2.1-19ubuntu1),
- gcc (= 4:10.2.0-1ubuntu1),
- gcc-10 (= 10.2.1-19ubuntu1),
- gcc-10-base (= 10.2.1-19ubuntu1),
- gcc-11-base (= 11-20210207-1ubuntu1),
+ findutils (= 4.8.0-1ubuntu1),
+ g++ (= 4:10.3.0-1ubuntu1),
+ g++-10 (= 10.3.0-1ubuntu1),
+ gcc (= 4:10.3.0-1ubuntu1),
+ gcc-10 (= 10.3.0-1ubuntu1),
+ gcc-10-base (= 10.3.0-1ubuntu1),
+ gcc-11-base (= 11.1.0-1ubuntu1~21.04),
  gettext (= 0.21-3ubuntu2),
  gettext-base (= 0.21-3ubuntu2),
  grep (= 3.6-1),
@@ -43,113 +44,114 @@
  hostname (= 3.23),
  init-system-helpers (= 1.60),
  intltool-debian (= 0.35.0+20060710.5),
- libacl1 (= 2.2.53-10),
+ libacl1 (= 2.2.53-10ubuntu1),
  libarchive-zip-perl (= 1.68-1),
- libasan6 (= 10.2.1-19ubuntu1),
- libatomic1 (= 11-20210207-1ubuntu1),
- libattr1 (= 1:2.4.48-6),
+ libasan6 (= 11.1.0-1ubuntu1~21.04),
+ libatomic1 (= 11.1.0-1ubuntu1~21.04),
+ libattr1 (= 1:2.4.48-6build1),
  libaudit-common (= 1:3.0-2ubuntu1),
  libaudit1 (= 1:3.0-2ubuntu1),
- libbinutils (= 2.36.1-0ubuntu1),
- libblkid1 (= 2.36.1-1ubuntu2),
- libbz2-1.0 (= 1.0.8-4ubuntu2),
- libc-bin (= 2.33-0ubuntu2),
- libc-dev-bin (= 2.33-0ubuntu2),
- libc6 (= 2.33-0ubuntu2),
- libc6-dev (= 2.33-0ubuntu2),
+ libbinutils (= 2.36.1-6ubuntu1),
+ libblkid1 (= 2.36.1-7ubuntu2),
+ libbz2-1.0 (= 1.0.8-4ubuntu3),
+ libc-bin (= 2.33-0ubuntu5),
+ libc-dev-bin (= 2.33-0ubuntu5),
+ libc6 (= 2.33-0ubuntu5),
+ libc6-dev (= 2.33-0ubuntu5),
  libcap-ng0 (= 0.7.9-2.2build1),
- libcap2 (= 1:2.44-1),
- libcc1-0 (= 11-20210207-1ubuntu1),
- libcom-err2 (= 1.45.7-1ubuntu1),
- libcrypt-dev (= 1:4.4.17-1ubuntu1),
- libcrypt1 (= 1:4.4.17-1ubuntu1),
- libctf-nobfd0 (= 2.36.1-0ubuntu1),
- libctf0 (= 2.36.1-0ubuntu1),
- libdb5.3 (= 5.3.28+dfsg1-0.6ubuntu3),
- libdebconfclient0 (= 0.256ubuntu1),
- libdebhelper-perl (= 13.3.3ubuntu2),
- libdpkg-perl (= 1.20.7.1ubuntu2),
- libelf1 (= 0.183-1),
+ libcap2 (= 1:2.44-1build1),
+ libcc1-0 (= 11.1.0-1ubuntu1~21.04),
+ libcom-err2 (= 1.45.7-1ubuntu2),
+ libcrypt-dev (= 1:4.4.17-1ubuntu3),
+ libcrypt1 (= 1:4.4.17-1ubuntu3),
+ libctf-nobfd0 (= 2.36.1-6ubuntu1),
+ libctf0 (= 2.36.1-6ubuntu1),
+ libdb5.3 (= 5.3.28+dfsg1-0.6ubuntu4),
+ libdebconfclient0 (= 0.256ubuntu3),
+ libdebhelper-perl (= 13.3.4ubuntu1),
+ libdpkg-perl (= 1.20.9ubuntu1),
+ libdw1 (= 0.183-8),
+ libelf1 (= 0.183-8),
  libfile-stripnondeterminism-perl (= 1.11.0-1),
- libgcc-10-dev (= 10.2.1-19ubuntu1),
- libgcc-s1 (= 11-20210207-1ubuntu1),
- libgcrypt20 (= 1.8.7-2ubuntu1),
+ libgcc-10-dev (= 10.3.0-1ubuntu1),
+ libgcc-s1 (= 11.1.0-1ubuntu1~21.04),
+ libgcrypt20 (= 1.8.7-2ubuntu2),
  libgdbm-compat4 (= 1.19-2),
  libgdbm6 (= 1.19-2),
- libgmp10 (= 2:6.2.1+dfsg-1ubuntu1),
- libgomp1 (= 11-20210207-1ubuntu1),
- libgpg-error0 (= 1.38-2),
+ libgmp10 (= 2:6.2.1+dfsg-1ubuntu2),
+ libgomp1 (= 11.1.0-1ubuntu1~21.04),
+ libgpg-error0 (= 1.38-2build1),
  libgssapi-krb5-2 (= 1.18.3-4),
- libicu67 (= 67.1-6ubuntu1),
- libisl23 (= 0.23-1),
- libitm1 (= 11-20210207-1ubuntu1),
+ libicu67 (= 67.1-6ubuntu2),
+ libisl23 (= 0.23-1build1),
+ libitm1 (= 11.1.0-1ubuntu1~21.04),
  libk5crypto3 (= 1.18.3-4),
  libkeyutils1 (= 1.6.1-2ubuntu1),
  libkrb5-3 (= 1.18.3-4),
  libkrb5support0 (= 1.18.3-4),
- liblz4-1 (= 1.9.3-1),
- liblzma5 (= 5.2.5-1.0),
+ liblz4-1 (= 1.9.3-1ubuntu0.1),
+ liblzma5 (= 5.2.5-1.0build2),
  libmagic-mgc (= 1:5.39-3),
  libmagic1 (= 1:5.39-3),
- libmount1 (= 2.36.1-1ubuntu2),
- libmpc3 (= 1.2.0-1),
- libmpfr6 (= 4.1.0-3),
+ libmount1 (= 2.36.1-7ubuntu2),
+ libmpc3 (= 1.2.0-1build1),
+ libmpfr6 (= 4.1.0-3build1),
  libnsl-dev (= 1.3.0-0ubuntu3),
  libnsl2 (= 1.3.0-0ubuntu3),
  libnspr4 (= 2:4.29-1),
  libnspr4-dev (= 2:4.29-1),
- libpam-modules (= 1.3.1-5ubuntu6),
- libpam-modules-bin (= 1.3.1-5ubuntu6),
- libpam-runtime (= 1.3.1-5ubuntu6),
- libpam0g (= 1.3.1-5ubuntu6),
- libpcre2-8-0 (= 10.36-2ubuntu1),
- libpcre3 (= 2:8.39-13),
- libperl5.32 (= 5.32.1-2),
+ libpam-modules (= 1.3.1-5ubuntu6.21.04.1),
+ libpam-modules-bin (= 1.3.1-5ubuntu6.21.04.1),
+ libpam-runtime (= 1.3.1-5ubuntu6.21.04.1),
+ libpam0g (= 1.3.1-5ubuntu6.21.04.1),
+ libpcre2-8-0 (= 10.36-2ubuntu5),
+ libpcre3 (= 2:8.39-13build3),
+ libperl5.32 (= 5.32.1-3ubuntu2),
  libpipeline1 (= 1.5.3-1),
- libseccomp2 (= 2.4.3-1ubuntu6),
- libselinux1 (= 3.1-3),
- libsigsegv2 (= 2.13-1),
- libsmartcols1 (= 2.36.1-1ubuntu2),
- libsqlite3-0 (= 3.34.1-2),
- libsqlite3-dev (= 3.34.1-2),
- libssl1.1 (= 1.1.1i-3ubuntu1),
- libstdc++-10-dev (= 10.2.1-19ubuntu1),
- libstdc++6 (= 11-20210207-1ubuntu1),
+ libseccomp2 (= 2.5.1-1ubuntu1),
+ libselinux1 (= 3.1-3build1),
+ libsigsegv2 (= 2.13-1ubuntu1),
+ libsmartcols1 (= 2.36.1-7ubuntu2),
+ libsqlite3-0 (= 3.34.1-3),
+ libsqlite3-dev (= 3.34.1-3),
+ libssl1.1 (= 1.1.1j-1ubuntu3.1),
+ libstdc++-10-dev (= 10.3.0-1ubuntu1),
+ libstdc++6 (= 11.1.0-1ubuntu1~21.04),
  libsub-override-perl (= 0.09-2),
- libsystemd0 (= 247.3-1ubuntu2),
- libtinfo6 (= 6.2+20201114-2),
- libtirpc-common (= 1.3.1-1),
- libtirpc-dev (= 1.3.1-1),
- libtirpc3 (= 1.3.1-1),
+ libsystemd0 (= 247.3-3ubuntu3.4),
+ libtinfo6 (= 6.2+20201114-2build1),
+ libtirpc-common (= 1.3.1-1build1),
+ libtirpc-dev (= 1.3.1-1build1),
+ libtirpc3 (= 1.3.1-1build1),
  libtool (= 2.4.6-15),
- libubsan1 (= 10.2.1-19ubuntu1),
+ libubsan1 (= 11.1.0-1ubuntu1~21.04),
  libuchardet0 (= 0.0.7-1),
- libudev1 (= 247.3-1ubuntu2),
+ libudev1 (= 247.3-3ubuntu3.4),
  libunistring2 (= 0.9.10-4),
- libuuid1 (= 2.36.1-1ubuntu2),
- libxml2 (= 2.9.10+dfsg-6.3build1),
- libzstd1 (= 1.4.8+dfsg-1),
- linux-libc-dev (= 5.10.0-14.15),
+ libuuid1 (= 2.36.1-7ubuntu2),
+ libxml2 (= 2.9.10+dfsg-6.3ubuntu0.1),
+ libzstd1 (= 1.4.8+dfsg-2build2),
+ linux-libc-dev (= 5.11.0-25.27),
  login (= 1:4.8.1-1ubuntu8),
  lsb-base (= 11.1.0ubuntu2),
- lto-disabled-list (= 1),
+ lto-disabled-list (= 7),
  m4 (= 1.4.18-5),
  make (= 4.3-4ubuntu1),
- man-db (= 2.9.4-1),
+ man-db (= 2.9.4-2),
  mawk (= 1.3.4.20200120-2),
- ncurses-base (= 6.2+20201114-2),
- ncurses-bin (= 6.2+20201114-2),
+ ncurses-base (= 6.2+20201114-2build1),
+ ncurses-bin (= 6.2+20201114-2build1),
  patch (= 2.7.6-7),
- perl (= 5.32.1-2),
- perl-base (= 5.32.1-2),
- perl-modules-5.32 (= 5.32.1-2),
+ perl (= 5.32.1-3ubuntu2),
+ perl-base (= 5.32.1-3ubuntu2),
+ perl-modules-5.32 (= 5.32.1-3ubuntu2),
  po-debconf (= 1.0.21+nmu1),
  rpcsvc-proto (= 1.4.2-0ubuntu4),
  sed (= 4.7-1ubuntu1),
  sensible-utils (= 0.0.14),
- sysvinit-utils (= 2.96-5ubuntu1),
- tar (= 1.33+dfsg-1),
- util-linux (= 2.36.1-1ubuntu2),
- xz-utils (= 5.2.5-1.0),
- zlib1g (= 1:1.2.11.dfsg-2ubuntu4),
- zlib1g-dev (= 1:1.2.11.dfsg-2ubuntu4)
+ sysvinit-utils (= 2.96-6ubuntu1),
+ tar (= 1.34+dfsg-1build1),
+ util-linux (= 2.36.1-7ubuntu2),
+ xz-utils (= 5.2.5-1.0build2),
+ zlib1g (= 1:1.2.11.dfsg-2ubuntu6),
+ zlib1g-dev (= 1:1.2.11.dfsg-2ubuntu6)

Revision history for this message

Paride Legovini (paride) wrote on 2021-07-21:

#11

My plan is now to:

- Setup a Groovy container
- Build nss 2:3.61-1ubuntu2 and verify the libnss3 is good
- Add Hirsute to sources.list and manually update the
Build-Deps, starting from the usual suspects (compilers),
hopefully finding which package breaks the dogtag-pki tests.

The testbed system will remain fixed (ubuntu:impish lxd container).

Revision history for this message

Paride Legovini (paride) wrote on 2021-07-21:

#12

The good news is that the test passes when building nss in a Groovy lxd container, but fails when copying that container (lxc copy), upgrading the copy to Hirsute and rebuilding there, so I have good pair of containers to do the "bisect" on.

Revision history for this message

Paride Legovini (paride) wrote on 2021-07-23:

#13

I tracked the problem down to the LTO optimizations that were enabled by default in dpkg 1.20.9ubuntu1.

Changed in nss (Ubuntu):
status:	New → Triaged
tags:	added: lto

Revision history for this message

In Mozilla Bugzilla #1721995, Paride Legovini (paride) wrote on 2021-07-23:

#16

Download full text (14.2 KiB)

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0

Steps to reproduce:

When compiling nss with LTO enabled (gcc -flto) on s390x the resulting libnss3 is not fully functional. I noticed this as the build causes a regression in the dogtag-pki tests which are part of the dogtag-pki Ubuntu package.

Newer releases of Ubuntu enable LTO by default when building packages. This specific issue will be worked around by disabling the optimizations specifically for this package and on s390x, however the problem is worth investigating upstream.

The error printout doesn't immediately point to optimization issues, however this is always reproducible, and reliably goes away by turning LTO off.

Steps to reproduce:
- Build nss on s390x with LTO enabled.
- Install dogtag-pki and ensure it uses the just built libnss3.
- Exercise the following tests: https://salsa.debian.org/freeipa-team/dogtag-pki/-/blob/master/debian/tests/pkispawn.

Actual results:

The tests fail:

autopkgtest [09:34:17]: test pkispawn: [-----------------------
>>>> IP address is 10.226.183.135
>>>> Hostname was:
>>>> /etc/hosts now has:
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
10.226.183.135 autopkgtest.debci autopkgtest
Starting installation...
Completed installation for pki-tomcat
Notice: Trust flag u is set automatically if the private key is present.
/usr/lib/python3/dist-packages/urllib3/connection.py:455: SubjectAltNameWarning: Certificate for autopkgtest.debci has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.)
  warnings.warn(
ERROR: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main
    scriptlet.spawn(deployer)
  File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 995, in spawn
    cert = deployer.setup_cert(client, tag)
  File "/usr/lib/python3/dist-packages/pki/server/deployment/__init__.py", line 355, in setup_cert
    return client.setupCert(request)
  File "/usr/lib/python3/dist-packages/pki/system.py", line 389, in setupCert
    response = self.connection.post(
  File "/usr/lib/python3/dist-packages/pki/client.py", line 55, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/pki/client.py", line 293, in post
    r = self.session.post(
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters...

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0

Steps to reproduce:

When compiling nss with LTO enabled (gcc -flto) on s390x the resulting libnss3 is not fully functional. I noticed this as the build causes a regression in the dogtag-pki tests which are part of the dogtag-pki Ubuntu package.

Newer releases of Ubuntu enable LTO by default when building packages. This specific issue will be worked around by disabling the optimizations specifically for this package and on s390x, however the problem is worth investigating upstream.

The error printout doesn't immediately point to optimization issues, however this is always reproducible, and reliably goes away by turning LTO off.

Steps to reproduce:
 - Build nss on s390x with LTO enabled.
 - Install dogtag-pki and ensure it uses the just built libnss3.
 - Exercise the following tests: https://salsa.debian.org/freeipa-team/dogtag-pki/-/blob/master/debian/tests/pkispawn.

Actual results:

The tests fail:

autopkgtest [09:34:17]: test pkispawn: [-----------------------
>>>> IP address is 10.226.183.135
>>>> Hostname was: 
>>>> /etc/hosts now has:
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
10.226.183.135 autopkgtest.debci autopkgtest
Starting installation...
Completed installation for pki-tomcat
Notice: Trust flag u is set automatically if the private key is present.
/usr/lib/python3/dist-packages/urllib3/connection.py:455: SubjectAltNameWarning: Certificate for autopkgtest.debci has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.)
  warnings.warn(
ERROR: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main
    scriptlet.spawn(deployer)
  File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 995, in spawn
    cert = deployer.setup_cert(client, tag)
  File "/usr/lib/python3/dist-packages/pki/server/deployment/__init__.py", line 355, in setup_cert
    return client.setupCert(request)
  File "/usr/lib/python3/dist-packages/pki/system.py", line 389, in setupCert
    response = self.connection.post(
  File "/usr/lib/python3/dist-packages/pki/client.py", line 55, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/pki/client.py", line 293, in post
    r = self.session.post(
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)

Loading deployment configuration from debian/tests/deploy.cfg.
Installation log: /var/log/pki/pki-ca-spawn.20210723093512.log
Installing CA into /var/lib/pki/pki-tomcat.

Installation failed: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

>>>> CA spawn failed:
2021-07-23 09:35:38 ERROR: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main
    scriptlet.spawn(deployer)
  File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 995, in spawn
    cert = deployer.setup_cert(client, tag)
  File "/usr/lib/python3/dist-packages/pki/server/deployment/__init__.py", line 355, in setup_cert
    return client.setupCert(request)
  File "/usr/lib/python3/dist-packages/pki/system.py", line 389, in setupCert
    response = self.connection.post(
  File "/usr/lib/python3/dist-packages/pki/client.py", line 55, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/pki/client.py", line 293, in post
    r = self.session.post(
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)

autopkgtest [09:35:38]: test pkispawn: -----------------------]
autopkgtest [09:35:39]: test pkispawn:  - - - - - - - - - - results - - - - - - - - - -
pkispawn             FAIL non-zero exit status 1
autopkgtest [09:35:39]: @@@@@@@@@@@@@@@@@@@@ summary
pkispawn             FAIL non-zero exit status 1

Expected results:

The tests all pass:

autopkgtest [22:28:02]: test pkispawn: [-----------------------
>>>> IP address is 10.226.183.148
>>>> Hostname was: 
>>>> /etc/hosts now has:
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
10.226.183.148 autopkgtest.debci autopkgtest
Starting installation...
Completed installation for pki-tomcat
Notice: Trust flag u is set automatically if the private key is present.
/usr/lib/python3/dist-packages/urllib3/connection.py:455: SubjectAltNameWarning: Certificate for autopkgtest.debci has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.)
  warnings.warn(
Loading deployment configuration from debian/tests/deploy.cfg.
Installation log: /var/log/pki/pki-ca-spawn.20210722222833.log
Installing CA into /var/lib/pki/pki-tomcat.

==========================================================================
                                INSTALLATION SUMMARY
    ==========================================================================

Administrator's username:             caadmin
      Administrator's PKCS #12 file:
            /root/.dogtag/pki-tomcat/ca_admin_cert.p12

To check the status of the subsystem:
            systemctl status pki-tomcatd@pki-tomcat.service

To restart the subsystem:
            systemctl restart pki-tomcatd@pki-tomcat.service

The URL for the subsystem is:
            https://autopkgtest.debci:8443/ca

PKI instances will be enabled upon system boot

==========================================================================

WARNING: Directory already exists: /etc/pki/pki-tomcat
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Loading deployment configuration from debian/tests/deploy.cfg.
Installation log: /var/log/pki/pki-kra-spawn.20210722222939.log
Installing KRA into /var/lib/pki/pki-tomcat.

==========================================================================
                                INSTALLATION SUMMARY
    ==========================================================================

Administrator's username:             kraadmin

To check the status of the subsystem:
            systemctl status pki-tomcatd@pki-tomcat.service

To restart the subsystem:
            systemctl restart pki-tomcatd@pki-tomcat.service

The URL for the subsystem is:
            https://autopkgtest.debci:8443/kra

PKI instances will be enabled upon system boot

==========================================================================

WARNING: Directory already exists: /etc/pki/pki-tomcat
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Loading deployment configuration from debian/tests/deploy.cfg.
Installation log: /var/log/pki/pki-ocsp-spawn.20210722223039.log
Installing OCSP into /var/lib/pki/pki-tomcat.

==========================================================================
                                INSTALLATION SUMMARY
    ==========================================================================

Administrator's username:             ocspadmin

To check the status of the subsystem:
            systemctl status pki-tomcatd@pki-tomcat.service

To restart the subsystem:
            systemctl restart pki-tomcatd@pki-tomcat.service

The URL for the subsystem is:
            https://autopkgtest.debci:8443/ocsp

PKI instances will be enabled upon system boot

==========================================================================

WARNING: Directory already exists: /etc/pki/pki-tomcat
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Loading deployment configuration from debian/tests/deploy.cfg.
Installation log: /var/log/pki/pki-tks-spawn.20210722223141.log
Installing TKS into /var/lib/pki/pki-tomcat.

==========================================================================
                                INSTALLATION SUMMARY
    ==========================================================================

Administrator's username:             tksadmin

To check the status of the subsystem:
            systemctl status pki-tomcatd@pki-tomcat.service

To restart the subsystem:
            systemctl restart pki-tomcatd@pki-tomcat.service

The URL for the subsystem is:
            https://autopkgtest.debci:8443/tks

PKI instances will be enabled upon system boot

==========================================================================

Loading deployment configuration from /var/lib/pki/pki-tomcat/tks/registry/tks/deployment.cfg.
Uninstallation log: /var/log/pki/pki-tks-destroy.20210722223248.log
Uninstalling TKS from /var/lib/pki/pki-tomcat.

Uninstallation complete.
WARNING: this 'OCSP' entry will NOT be deleted from security domain 'debci Security Domain'!
WARNING: security domain 'debci Security Domain' may be offline or unreachable!
ERROR: subprocess.CalledProcessError:  Command '['/usr/bin/sslget', '-n', 'subsystemCert cert-pki-tomcat', '-p', '~EGeO^i!Ai4^', '-d', '/etc/pki/pki-tomcat/alias', '-e', 'name="/var/lib/pki/pki-tomcat"&type=OCSP&list=ocspList&host=autopkgtest.debci&sport=8443&ncsport=8443&adminsport=8443&agentsport=8443&operation=remove', '-v', '-r', '/ca/agent/ca/updateDomainXML', 'autopkgtest.debci:8443']' returned non-zero exit status 6.!
Loading deployment configuration from /var/lib/pki/pki-tomcat/ocsp/registry/ocsp/deployment.cfg.
Uninstallation log: /var/log/pki/pki-ocsp-destroy.20210722223255.log
Uninstalling OCSP from /var/lib/pki/pki-tomcat.

Uninstallation complete.
ERROR: unable to access security domain. Continuing .. HTTPSConnectionPool(host='autopkgtest.debci', port=8443): Max retries exceeded with url: /ca/rest/securityDomain/domainInfo (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x3ff8af5a2b0>: Failed to establish a new connection: [Errno 111] Connection refused')) 
WARNING: this 'KRA' entry will NOT be deleted from security domain 'debci Security Domain'!
WARNING: security domain 'debci Security Domain' may be offline or unreachable!
ERROR: subprocess.CalledProcessError:  Command '['/usr/bin/sslget', '-n', 'subsystemCert cert-pki-tomcat', '-p', '~EGeO^i!Ai4^', '-d', '/etc/pki/pki-tomcat/alias', '-e', 'name="/var/lib/pki/pki-tomcat"&type=KRA&list=kraList&host=autopkgtest.debci&sport=8443&ncsport=8443&adminsport=8443&agentsport=8443&operation=remove', '-v', '-r', '/ca/agent/ca/updateDomainXML', 'autopkgtest.debci:8443']' returned non-zero exit status 6.!
Loading deployment configuration from /var/lib/pki/pki-tomcat/kra/registry/kra/deployment.cfg.
Uninstallation log: /var/log/pki/pki-kra-destroy.20210722223257.log
Uninstalling KRA from /var/lib/pki/pki-tomcat.

Uninstallation complete.
WARNING: this 'CA' entry will NOT be deleted from security domain 'debci Security Domain'!
WARNING: security domain 'debci Security Domain' may be offline or unreachable!
ERROR: subprocess.CalledProcessError:  Command '['/usr/bin/sslget', '-n', 'subsystemCert cert-pki-tomcat', '-p', '~EGeO^i!Ai4^', '-d', '/etc/pki/pki-tomcat/alias', '-e', 'name="/var/lib/pki/pki-tomcat"&type=CA&list=caList&host=autopkgtest.debci&sport=8443&ncsport=8443&adminsport=8443&agentsport=8443&operation=remove', '-v', '-r', '/ca/agent/ca/updateDomainXML', 'autopkgtest.debci:8443']' returned non-zero exit status 6.!
Loading deployment configuration from /var/lib/pki/pki-tomcat/ca/registry/ca/deployment.cfg.
Uninstallation log: /var/log/pki/pki-ca-destroy.20210722223258.log
Uninstalling CA from /var/lib/pki/pki-tomcat.

Uninstallation complete.
>>>> All done!
autopkgtest [22:32:59]: test pkispawn: -----------------------]
autopkgtest [22:33:00]: test pkispawn:  - - - - - - - - - - results - - - - - - - - - -
pkispawn             PASS
autopkgtest [22:33:00]: @@@@@@@@@@@@@@@@@@@@ summary
pkispawn             PASS

Revision history for this message

Paride Legovini (paride) wrote on 2021-07-23:

#14

All of the above still applies to nss 3.68-1, for which I'm preparing a merge right now.

Revision history for this message

Paride Legovini (paride) wrote on 2021-07-23:

#15

MP for a merge from Debian which also disabled LTO via DEB_BUILD_MAINT_OPTIONS=optimize=-lto:

https://code.launchpad.net/~paride/ubuntu/+source/nss/+git/nss/+merge/406163

summary:

- Test of dogtag-pki is failing on s390x vs the nss v3.63 in impish-
- proposed
+ Test of dogtag-pki is failing on s390x due to LTO

Revision history for this message

In Mozilla Bugzilla #1721995, Rrelyea (rrelyea) wrote on 2021-07-28:

#17

Evidently there is a similiar issue in fedora: https://bugzilla.redhat.com/show_bug.cgi?id=1986627

Paride Legovini (paride) on 2021-08-08

tags:	removed: server-next update-excuse
affects:	fedora → nss (Fedora)
Changed in nss (Ubuntu):
assignee:	Paride Legovini (paride) → nobody

Revision history for this message

In Mozilla Bugzilla #1721995, Bbeurdouche (bbeurdouche) wrote on 2021-08-10:

#18

Bob, I am marking this P3 for now as this is not a supported platform for us, but feel free to update the priority.

Revision history for this message

In Mozilla Bugzilla #1721995, Rrelyea (rrelyea) wrote on 2021-08-10:

#19

I did a scratch build of nss with LTO on in fedora, so the tests were working correctly. I haven't tested it against dogtag yet. Once NSS 3.69 builds are complete, I'll drop the LTO changes into fedora and see if our dogtag team has any issues.

Bug Watch Updater (bug-watch-updater) on 2021-08-20

Changed in nss:
status:	Unknown → New

Revision history for this message

Paride Legovini (paride) wrote on 2022-02-18:

#20

I reviewed this issue again and I don't think we're ready to drop the "Disable LTO on s390x" delta for now. There are newer NSS upstream versions worth testing but in Ubuntu we're still at 3.68 for now.

Revision history for this message

Athos Ribeiro (athos-ribeiro) wrote on 2022-02-28:

#21

We decided to stay in 3.68.x (which is an ESR release) for jammy because it is an LTS release.

We could revisit this once we merge a newer NSS version next cycle, as suggested by Paride.

	Status	Importance	Assigned to
NSS	New	Unknown	mozilla-bugs #1721995
nss (Fedora)	Unknown	Unknown	redhat-bugs #1986627
nss (Ubuntu)	Triaged	Undecided	Unassigned

Ubuntu
nss package

Test of dogtag-pki is failing on s390x due to LTO

Bug Description

Other bug subscribers

Remote bug watches

Ubuntunss package

Test of dogtag-pki is failing on s390x due to LTO

Bug Description

Other bug subscribers

Remote bug watches

Ubuntu
nss package