samtools-legacy misses 'Remove compressBound assertions' (PR #1258) fix

Bug #1983255 reported by Frank Heimes
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
Undecided
Unassigned
libbio-samtools-perl (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned
Kinetic
Fix Released
Undecided
Unassigned
samtools-legacy (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned
Kinetic
Fix Released
High
Unassigned

Bug Description

SRU Justification:
==================

[Impact]

 * zlib requires a fix for 'compressBound() returns an incorrect result
   on z15' (described at LP#1961427 in more detail).

 * But the above patch expects that another fix 'Remove compressBound
   assertions' (PR #1258) is in place for all bgzf.c implementations.

 * The packages htslib and bedtools are covered in LP#1961427.

 * Unfortunately more 'embedded' or better 'bundled' bgzf.c implementations
   exist, that are difficult to find upfront, hence the zlib proposed migration
   now pointed to libbio-samtools-perl.

 * libbio-samtools-perl is linked to libbam-dev from samtools-legacy;
   and samtools-legacy contains another bgzf.c implementation.

 * Hence PR #1258 is needed to fix bgzf.c as part of samtools-legacy.

 * And libbio-samtools-perl needs then rebuild to like against the
   fixed libbam-dev.

 * PR #1258 removes the Assert lines and modifies bgzf_compress.

[Test Plan]

* An IBM zSystem (LPAR, z/VM guest or KVM virtual machine)
  with Ubuntu Server 20.04 (or 22.04 or 22.10).

* Having the zlib version from -proposed installed or from PPA:
  ppa:fheimes/lp1961427

* The easiest way to test this is based on the autopkgtest of the affected
  package, here 'samtools-legacy' (here a manual autopkgtest execution):
      autopkgtest --no-built-binaries --apt-pocket=proposed --apt-upgrade \
      --shell-fail --setup-commands="apt -y -q install \
      software-properties-common; apt-key adv --keyserver \
      hkp://keyserver.ubuntu.com:80 --recv-keys \
      73E9E91F16C43C45C621AE3EDF0A28DEFAFEB468; add-apt-repository -y \
      ppa:fheimes/lp1983255; apt update; apt -y upgrade; apt install -y \
      libbam-dev=0.1.19+dfsg-5ubuntu1" samtools-legacy_0.1.19+dfsg-5ubuntu1.dsc \
      -- lxd autopkgtest/ubuntu/kinetic/s390x
      (for focal and jammy adjust the versions accordingly)

 * Without PR #1258 in place, the error: 'bgzf.c:<line-number>: bgzf_open:
   Assertion `compressBound(BGZF_BLOCK_SIZE) < BGZF_MAX_BLOCK_SIZE'
   will occur.

 * If the above test is successful one can rebuild 'libbio-samtools-perl',
   and run the autopkgtest of 'libbio-samtools-perl' on top.

[Where problems could occur]

 * The bgzf.c implementation as it is in 'samtools-legacy' is very outdated
   and the code was not touched for close to 10 years.

 * Hence PR #1258 could not directly be applied, and a backport was needed.

 * Issues could be that checks (due to missing asserts) are now insufficient,

 * or (un-)compress does not work properly anymore, due to the changes in
   bgzf_compress.

 * But the autopkgtest(s) will be helpful to ensure proper functionality.

[Other Info]

 * The different bgzf.c versions that are bundled are almost all
   slightly different implementations, hence the backport of PR #1258
   needs to be a bit adjusted here and there.
__________

samtools-legacy is the source package for libbam-dev,
which is used by libbio-samtools-perl (autopkgtests),
that is for example triggered by a zlib update (proposed-transition).

zlib requires a fix for 'compressBound() returns an incorrect result on z15' LP#1961427 that again requires the 'Remove compressBound assertions' (PR #1258) fix in all bgzf.c implementations, like htslib, bedtools or even samtools-legacy.

Problematic is that the code differs quite a bit from the one where the PR#1258 was designed for and that is used in the htslib or bedtools package (no bgzf_hopen - looks a bit out of date), so it does not apply cleanly.

According to upstream 'https://github.com/lh3/samtools-legacy' the project
is "For testing only." and marked as "DON'T USE!" and it's last update
is from Nov 9, 2013.

Frank Heimes (fheimes)
description: updated
Revision history for this message
Frank Heimes (fheimes) wrote (last edit ):

I've created a backport of PR #1258 that applied to bgzf.c from samtools-legacy.
I did a PPA test build: https://launchpad.net/~fheimes/+archive/ubuntu/lp1983255

Changed in samtools-legacy (Ubuntu):
status: New → In Progress
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "debdiff_kinetic_samtools-legacy_from_0.1.19+dfsg-5_to_0.1.19+dfsg-5ubuntu1.diff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Frank Heimes (fheimes) wrote :
Revision history for this message
Olivier Gayot (ogayot) wrote :

Hi @fheimes!
I stumbled upon this bug when looking at proposed-migrations (i.e., zlib vs libbio-samtools-perl).

I ran autopkgtest on s390x against proposed and pinned libbam-dev to the version in your ppa (https://launchpad.net/~fheimes/+archive/ubuntu/lp1983255) but unfortunately the test is still red for me.

Would you mind having a quick glance and let me know if you think I overlooked or missed something?

I'm attaching the logs.

Best regards,
Olivier

Revision history for this message
Frank Heimes (fheimes) wrote :

It turned out that libbio-samtools-perl is statically linked to libbam-dev, hence libbio-samtools-perl needs a rebuild, too.
So with these two it seems to work:
ppa:fheimes/samtools-legacy
ppa:fheimes/libbio-samtools-perl
My local autopkgtest runs fine with these. (also thx to Olivier)

Revision history for this message
Olivier Gayot (ogayot) wrote :

With @fheimes's help, I was able to run autopkgtest successfully.
We figured that libbio-samtools-perl requires a rebuild since it statically links against libbam-dev.

The autopkgtest suite runs like a charm after a rebuild.

Also, submitting the right .debdiff for samtools-legacy with Frank's permission. The original .debdiff does not apply on a clean source tree.

Thanks!

Revision history for this message
Frank Heimes (fheimes) wrote :

It's a similar situation with the zlib (1:1.2.11.dfsg-2ubuntu1.4)) proposed-migration in focal:
https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#zlib

Again focal's version of the samtools-legacy package needed the fix 'PR #1258'.
And based on that, libbio-samtools-perl requires a rebuild since it statically links against libbam-dev.

See PPA: https://launchpad.net/~fheimes/+archive/ubuntu/lp1983255

debdiffs below ...

Revision history for this message
Frank Heimes (fheimes) wrote :
Revision history for this message
Frank Heimes (fheimes) wrote :
tags: added: focal
Frank Heimes (fheimes)
description: updated
Revision history for this message
Frank Heimes (fheimes) wrote :

Subscribing ubuntu-sponsors for sponsoring the SRU for focal....

Frank Heimes (fheimes)
description: updated
Revision history for this message
Lukas Märdian (slyon) wrote :

Thanks Frank and Olivier for the investigations and debdiff.

LGTM. I've uploaded your patch into Kinetic.
I did fix the "Origin: ..." patch header, as it contained a broken URL. Also, I added another "Bug-Ubuntu: ..." header, to reference this bug report, too.

Lastly, I think this patch is worth upstreaming to Debian, could you please use "submittodebian" to send it their way, so we can hopefully get the package back in sync soon? Thanks!

Changed in samtools-legacy (Ubuntu Kinetic):
status: In Progress → Fix Committed
Revision history for this message
Olivier Gayot (ogayot) wrote :

Adding no-change rebuild .debdiff for libbio-samtools-perl (kinetic). To be uploaded once samtools-legacy has finished building. Thanks!

Revision history for this message
Lukas Märdian (slyon) wrote :

I adopted the version string of the kinetic debdiff and uploaded it into Jammy, which contains the same version of samtools-legacy.

Changed in samtools-legacy (Ubuntu Jammy):
status: New → In Progress
Revision history for this message
Lukas Märdian (slyon) wrote :

Uploaded the Focal debdiff as well, after applying the same fixes (patch header cleanup) as mentioned in the #11 above.

Changed in samtools-legacy (Ubuntu Focal):
status: New → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package samtools-legacy - 0.1.19+dfsg-5ubuntu1

---------------
samtools-legacy (0.1.19+dfsg-5ubuntu1) kinetic; urgency=medium

  * Add backport of upstream commit 515f6df as
    d/p/515f6df-Remove-compressBound-assertions-PR-1258.patch
    in support of fixing 'zlib: compressBound() returns an incorrect result
    on z15'. (LP: #1983255)

 -- Frank Heimes <email address hidden> Mon, 01 Aug 2022 19:03:43 +0200

Changed in samtools-legacy (Ubuntu Kinetic):
status: Fix Committed → Fix Released
Simon Chopin (schopin)
Changed in libbio-samtools-perl (Ubuntu Kinetic):
status: New → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libbio-samtools-perl - 1.43-3build4

---------------
libbio-samtools-perl (1.43-3build4) kinetic; urgency=medium

  * No-change rebuild against samtools-legacy (LP: #1983255)

 -- Olivier Gayot <email address hidden> Fri, 12 Aug 2022 11:33:17 +0200

Changed in libbio-samtools-perl (Ubuntu Kinetic):
status: Fix Committed → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Frank, or anyone else affected,

Accepted samtools-legacy into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/samtools-legacy/0.1.19+dfsg-5ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in samtools-legacy (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-jammy
Changed in samtools-legacy (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Steve Langasek (vorlon) wrote :

Hello Frank, or anyone else affected,

Accepted samtools-legacy into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/samtools-legacy/0.1.19-4ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Frank Heimes (fheimes) wrote :

Tested 'samtools-legacy' with the help of a (no-change) rebuild on jammy 'libbio-samtools-perl_1.43-3build3' that makes use of 'libbam-dev_0.1.19+dfsg-5ubuntu0.1' (taken from proposed), which is the binary package of 'samtools-legacy'.
(Notice that 'libbio-samtools-perl' is the package tha is causing the zlib proposed migration regression.)

$ autopkgtest-build-lxd images:ubuntu/jammy/s390x
Using http://squid.internal:3128 as container proxy
Creating autopkgtest-...
$

$ autopkgtest --no-built-binaries --apt-pocket=proposed --apt-upgrade --shell-fail --setup-commands="apt -y -q install software-properties-common; add-apt-repository -y 'deb http://us.ports.ubuntu.com/ubuntu-ports/ jammy-proposed main universe'; apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 73E9E91F16C43C45C621AE3EDF0A28DEFAFEB468; add-apt-repository -y ppa:fheimes/libbio-samtools-perl; apt -y update; apt install -y libbam-dev=0.1.19+dfsg-5ubuntu0.1; apt install -y libbio-samtools-perl=1.43-3build3" libbio-samtools-perl_1.43-3build3.dsc -- lxd autopkgtest/ubuntu/jammy/s390x
...
All tests successful.
Files=1, Tests=4, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.38 cusr 0.07 csys = 0.47 CPU)
Result: PASS
autopkgtest [13:43:04]: test autodep8-perl-recommends: -----------------------]
autopkgtest [13:43:05]: test autodep8-perl-recommends: - - - - - - - - - - results - - - - - - - - - -
autodep8-perl-recommends PASS (superficial)
autopkgtest [13:43:05]: @@@@@@@@@@@@@@@@@@@@ summary
autodep8-perl-build-deps PASS
autodep8-perl PASS (superficial)
autodep8-perl-recommends PASS (superficial)
$

Hence updating tags accordingly to verification-done-jammy.

tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Frank Heimes (fheimes) wrote :

Tested 'samtools-legacy' with the help of a (no-change) rebuild 'libbio-samtools-perl-1.43-2build2.dsc' on focal that makes use of 'libbam-dev_0.1.19-4ubuntu0.1' (taken from proposed), which is the binary package of 'samtools-legacy'.
(Notice that 'libbio-samtools-perl' is the package tha is causing the zlib proposed migration regression.)

$ autopkgtest-build-lxd images:ubuntu/focal/s390x
Using http://squid.internal:3128 as container proxy
Creating autopkgtest-prepare-...
$

$ autopkgtest --no-built-binaries --apt-pocket=proposed --apt-upgrade --shell-fail --setup-commands="apt -y -q install software-properties-common; add-apt-repository -y 'deb http://us.ports.ubuntu.com/ubuntu-ports/ focal-proposed main universe'; apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 73E9E91F16C43C45C621AE3EDF0A28DEFAFEB468; add-apt-repository -y ppa:fheimes/libbio-samtools-perl; apt -y update; apt install -y libbam-dev=0.1.19-4ubuntu0.1; apt install -y libbio-samtools-perl=1.43-2build2" libbio-samtools-perl_1.43-2build2.dsc -- lxd autopkgtest/ubuntu/focal/s390x
...
All tests successful.
Files=1, Tests=4, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.39 cusr 0.07 csys = 0.48 CPU)
Result: PASS
autopkgtest [15:43:37]: test autodep8-perl-recommends: -----------------------]
autopkgtest [15:43:38]: test autodep8-perl-recommends: - - - - - - - - - - results - - - - - - - - - -
autodep8-perl-recommends PASS (superficial)
autopkgtest [15:43:38]: @@@@@@@@@@@@@@@@@@@@ summary
autodep8-perl-build-deps PASS
autodep8-perl PASS (superficial)
autodep8-perl-recommends PASS (superficial)
$

Hence updating tags accordingly to verification-done-focal.

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Frank Heimes (fheimes) wrote :

Adding no-change rebuild debdiff for libbio-samtools-perl (focal).
To be uploaded once samtools-legacy (focal) is done.

Revision history for this message
Frank Heimes (fheimes) wrote :

Adding no-change rebuild debdiff for libbio-samtools-perl (jammy).
To be uploaded once samtools-legacy (jammy) is done.

Revision history for this message
Lukas Märdian (slyon) wrote (last edit ):

I fixed the version number -3build3 => -3build2.1 for Jammy, as the -3build3 version string already exists in the archive. I've added a bug reference to this bug into the changelog, too.

Uploaded into Jammy (unapproved queue).

Changed in libbio-samtools-perl (Ubuntu Jammy):
status: New → In Progress
Revision history for this message
Lukas Märdian (slyon) wrote (last edit ):

Version number -2build2 is still fine for Focal, as that is smaller than what we have in Jammy (-3build2). But I've added a bug reference to this bug into the changelog.

Uploaded into Focal (unapproved queue).

Changed in libbio-samtools-perl (Ubuntu Focal):
status: New → In Progress
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Frank, or anyone else affected,

Accepted libbio-samtools-perl into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libbio-samtools-perl/1.43-2build2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in libbio-samtools-perl (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
removed: verification-done verification-done-focal
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Frank, or anyone else affected,

Accepted libbio-samtools-perl into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libbio-samtools-perl/1.43-3build2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in libbio-samtools-perl (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed-jammy
removed: verification-done-jammy
Revision history for this message
Frank Heimes (fheimes) wrote :

successfully verified on jammy based on libbio-samtools-perl test that uses libbam-dev from samtools legacy (updating tag accordingly)

tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package samtools-legacy - 0.1.19+dfsg-5ubuntu0.1

---------------
samtools-legacy (0.1.19+dfsg-5ubuntu0.1) jammy; urgency=medium

  * Add backport of upstream commit 515f6df as
    d/p/515f6df-Remove-compressBound-assertions-PR-1258.patch
    in support of fixing 'zlib: compressBound() returns an incorrect result
    on z15'. (LP: #1983255)

 -- Frank Heimes <email address hidden> Mon, 01 Aug 2022 19:03:43 +0200

Changed in samtools-legacy (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libbio-samtools-perl - 1.43-3build2.1

---------------
libbio-samtools-perl (1.43-3build2.1) jammy; urgency=medium

  * No-change rebuild for the samtools-legacy/libbam-dev update. (LP: #1983255)

 -- Frank Heimes <email address hidden> Sat, 20 Aug 2022 13:38:25 +0200

Changed in libbio-samtools-perl (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for samtools-legacy has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Frank Heimes (fheimes) wrote :

I now also successfully re-did the verification on focal,
hence updating the verification tag accordingly.

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package samtools-legacy - 0.1.19-4ubuntu0.1

---------------
samtools-legacy (0.1.19-4ubuntu0.1) focal; urgency=medium

  * Add backport of upstream commit 515f6df as
    d/p/515f6df-Remove-compressBound-assertions-PR-1258.patch
    in support of fixing 'zlib: compressBound() returns an incorrect result
    on z15'. (LP: #1983255)

 -- Frank Heimes <email address hidden> Mon, 08 Aug 2022 13:28:37 +0200

Changed in samtools-legacy (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libbio-samtools-perl - 1.43-2build2

---------------
libbio-samtools-perl (1.43-2build2) focal; urgency=medium

  * No-change rebuild for the samtools-legacy/libbam-dev update. (LP: #1983255)

 -- Frank Heimes <email address hidden> Sat, 20 Aug 2022 16:15:00 +0200

Changed in libbio-samtools-perl (Ubuntu Focal):
status: Fix Committed → Fix Released
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: New → Fix Released
To post a comment you must log in.