Ubuntu
autofs package

autofs: regression on focal->jammy upgrade: automount daemon hangs

Bug #1982219 reported by James Dingwall
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
autofs (Ubuntu)
Fix Released
Undecided
Christian Ehrhardt 
Jammy
Fix Released
Undecided
Christian Ehrhardt 
Kinetic
Fix Released
Undecided
Christian Ehrhardt 

Bug Description

[Impact]

 * There is a locking issue in v5.1.8 of autofs where unbalanced
   locking leads to a hang with certain operations.

 * Fix was proposed and acknowledged upstream (see below)

 * Backport that fix to releases with 5.1.8 (= Jammy and Kinetic)

[Test Plan]

 * This needs a rather complex setup, but gladly the users/community
   is rather active and responsive on this. So we will describe the test,
   but rely on them to verify the fix (pre-checked on PPA,
   was good there).

 * Setup: Use automount with maps held in a Samba active directory
   (or elsewhere, but so they have to be fetched). Autofs then needs
   to be configured to use kerberos authentication (that is what
   brings the cyrus-sasl code in) to connect using LDAP to retrieve
   the maps.
   Without the fix during the retrieval of the master map the daemon hangs
   and makes no progress.
   -m = list information about the configured automounter maps, then exit
   -f = Run the daemon in the foreground and log to stderr
   Good: `automount -m -f` dumps the configured maps and the daemon
   starts correctly and manages the expect mounts correctly.

   Bad: `automount -m -f` hangs

[Where problems could occur]

 * Of the many things autofs can do this code change is only
   in cyrus-sasl. So one can expect potential impact to be only
   in that functionality - others should see no change at all.
   Furthermore due to the nature of the change (locking) what
   you'd expect as regression would be hangs or uncontrolled
   concurrent access.

[Other Info]

 * Upstream releases slowly and rarely, so the patch is
   acknowledged but not yet in git or a release. But that
   is as much as we can get for now.

--- original report ---

I use automount maps held in a Samba active directory. Autofs is configured to use kerberos authentication to connect using LDAP to retrieve the maps. During the retrieval of the master map the daemon hangs and makes no progress.

# apt-cache policy autofs
autofs:
  Installed: 5.1.8-1ubuntu1.1
  Candidate: 5.1.8-1ubuntu1.1
  Version table:
 *** 5.1.8-1ubuntu1.1 500
        500 http://gb.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     5.1.8-1ubuntu1 500
        500 http://gb.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04 LTS
Release: 22.04
Codename: jammy

After adding some debugging I have traced this to an unbalanced pthread_mutex_lock/unlock which appears to have been introduced in d0d99cfc0737f34af326ba3f8f35f45625300576 autofs-5.1.6 - fix ldap sasl reconnect problem. The attached patch which performs a pthread_mutex_unlock() in the alternate success path of sasl_do_kinit_ext_cc() resolves the problem for me. I intend to post this to <email address hidden> once my membership to the list is accepted.

List post: https://marc.info/?l=autofs&m=165826158526742&w=2

See original description

Related branches

~paelzer/ubuntu/+source/autofs:lp-1982219-lock-imbalance-KINETIC
git-ubuntu bot: Approve
Andreas Hasenack: Approve
Canonical Server Reporter: Pending requested
Diff: 75 lines (+53/-0)
3 files modified
debian/changelog (+7/-0)
debian/patches/autofs-5.1.8-ldap-kerberos-leads-to-automount-hang-p.patch (+45/-0)
debian/patches/series (+1/-0)
~paelzer/ubuntu/+source/autofs:lp-1982219-lock-imbalance-JAMMY
git-ubuntu bot: Approve
Andreas Hasenack: Approve
Canonical Server Reporter: Pending requested
Diff: 75 lines (+53/-0)
3 files modified
debian/changelog (+7/-0)
debian/patches/autofs-5.1.8-ldap-kerberos-leads-to-automount-hang-p.patch (+45/-0)
debian/patches/series (+1/-0)
Revision history for this message
James Dingwall (a-james-launchpad) wrote :
James Dingwall (a-james-launchpad)
description: updated
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "sasl_do_kinit_ext_cc-unbalanced-lock.diff" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Revision history for this message
rdratlos (rdratlos) wrote :

I can confirm that above patch solves the issue.
As this is a showstopper for LTS I would like to ask maintainers to provide an update of autofs package.
Kernel developers have noted the patch but are not very willing to provide an immediate correction.

Revision history for this message
rdratlos (rdratlos) wrote :

This issue also affects autofs 5.1.7.
Last working release is currently 5.1.6.

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thanks for the bug report.

I don't have time to reproduce this problem right now (the setup seems complex from what you describe), but I saw you post upstream and their reply. Since they've acknowledged there's indeed a bug, I am tagging this as Triaged and adding the server-todo tag.

Thanks.

Changed in autofs (Ubuntu Jammy):
status: New → Triaged
Changed in autofs (Ubuntu Kinetic):
status: New → Triaged
tags: added: server-todo
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

The upstream acceptance of this seems to be ok, but delayed.
But given that it is acknowledged and queued for acceptance - as well as breaking major use cases we should add it to the package indeed.

I'll prepare PPAs and MRs for this - but due to the complexity I'd look to James and rdratlos to maybe help us testing this once the time for an SRU verification of this is around.

PPA: https://launchpad.net/~paelzer/+archive/ubuntu/lp-1982219-autofs-lock-imbalance
MR-Jammy: https://code.launchpad.net/~paelzer/ubuntu/+source/autofs/+git/autofs/+merge/427536
MR-Kinetic: https://code.launchpad.net/~paelzer/ubuntu/+source/autofs/+git/autofs/+merge/427537

Changed in autofs (Ubuntu Jammy):
assignee: nobody → Christian Ehrhardt  (paelzer)
Changed in autofs (Ubuntu Kinetic):
assignee: nobody → Christian Ehrhardt  (paelzer)
Revision history for this message
James Dingwall (a-james-launchpad) wrote :

I did a `dpkg -i` of the autofs and autofs-ldap .deb from the PPA:

$ apt-cache policy autofs
autofs:
  Installed: 5.1.8-1ubuntu1.2~jammyppa1
  Candidate: 5.1.8-1ubuntu1.2~jammyppa1
  Version table:
 *** 5.1.8-1ubuntu1.2~jammyppa1 100
        100 /var/lib/dpkg/status
     5.1.8-1ubuntu1.1 500
        500 http://gb.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     5.1.8-1ubuntu1 500
        500 http://gb.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

These new packages resolve the problem I was having, `automount -m -f` dumps the configured maps without hanging and the daemon starts correctly and manages the expect mounts correctly.

Thank you:)

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks James for the quick check, that provides the confidence to go on with this.

Christian Ehrhardt  (paelzer)
description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Review was ok - I now uploaded this to Kinetic and Jammy - on the latter it will enter the SRU queue and soon hopefully they will accept it into jammy-proposed from where James can then test it.

Changed in autofs (Ubuntu Jammy):
status: Triaged → In Progress
Changed in autofs (Ubuntu Kinetic):
status: Triaged → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package autofs - 5.1.8-1ubuntu3

---------------
autofs (5.1.8-1ubuntu3) kinetic; urgency=medium

  * d/p/autofs-5.1.8-ldap-kerberos-leads-to-automount-hang-p.patch: fix
    lock imbalance (LP: #1982219)

 -- Christian Ehrhardt <email address hidden> Thu, 28 Jul 2022 07:27:10 +0200

Changed in autofs (Ubuntu Kinetic):
status: Fix Committed → Fix Released
Revision history for this message
rdratlos (rdratlos) wrote :

Christian's bugfix package works perfectly on Jammy. As it is a pre-requisite to solve bug #1984073 I would recommend to backport the solution to Jammy, too.

Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello James, or anyone else affected,

Accepted autofs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autofs/5.1.8-1ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in autofs (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
James Dingwall (a-james-launchpad) wrote :

I have installed the .deb packages from proposed and these packages are working for me.

# apt-cache policy autofs
autofs:
  Installed: 5.1.8-1ubuntu1.2
  Candidate: 5.1.8-1ubuntu1.2
  Version table:
 *** 5.1.8-1ubuntu1.2 100
        100 /var/lib/dpkg/status
     5.1.8-1ubuntu1.1 500
        500 http://gb.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     5.1.8-1ubuntu1 500
        500 http://gb.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

Lucas Kanashiro (lucaskanashiro)
tags: added: verification-done verification-done-jammy
removed: verification-needed verification-needed-jammy
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package autofs - 5.1.8-1ubuntu1.2

---------------
autofs (5.1.8-1ubuntu1.2) jammy; urgency=medium

  * d/p/autofs-5.1.8-ldap-kerberos-leads-to-automount-hang-p.patch: fix
    lock imbalance (LP: #1982219)

 -- Christian Ehrhardt <email address hidden> Thu, 28 Jul 2022 07:29:32 +0200

Changed in autofs (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Robie Basak (racb) wrote : Update Released

The verification of the Stable Release Update for autofs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.