autofs fails to mount nfs4 shares with "error 0x3 getting portmap client"

Bug #1970264 reported by Michael Kiermaier
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
autofs (Ubuntu)
Fix Released
High
Sergio Durigan Junior
Jammy
Fix Released
High
Sergio Durigan Junior
Kinetic
Fix Released
High
Sergio Durigan Junior

Bug Description

[ Impact ]

NFSv4-only shares cannot be automounted if the server isn't running (or is blocking) rpcbind, even though NFSv4 doesn't require rpcbind anymore.

[ Test Plan ]

You will two Jammy VMs (unfortunately lxd VMs won't work, so you will likely need to use qemu/libvirt). One will act as the NFSv4-only server, and the other will be the client with autofs configured.

Here are the steps to configure the server VM:

Install the NFS server:

# apt install -y nfs-kernel-server

Then, edit the nfs-server.service file:

# systemctl edit --full nfs-server.service

Remove any mentions to rpcbind.socket from the [Unit] section. Also, edit the ExecStart line to look like:

ExecStart=/usr/sbin/rpc.nfsd --no-nfs-version 3

Stop and mask the rpcbind.* systemd units:

# systemctl stop rpcbind.socket rpcbind.service
# systemctl mask rpcbind.socket rpcbind.service

Create a directory to share and export it:

# mkdir /testshare
# touch /testshare/testfile
# echo -e '/testshare\t*(rw,sync,no_subtree_check)' >> /etc/exports

Restart nfs-server:

# systemctl restart nfs-server.service

Here are the steps to configure the client VM:

Install nfs-common and autofs:

# apt install -y nfs-common autofs

Add the automount configuration for the share (replace NFS_SERVER_IP with the IP address of the server VM you've just configured above):

# echo -e '/mnt\t/etc/auto.nfs' >> /etc/auto.master
# echo -e 'test\t-fstype=nfsv4\tNFS_SERVER_IP:/testshare' >> /etc/auto.nfs

Restart autofs:

# systemctl restart autofs.service

Try to access the share and notice that it won't automount:

# ls /mnt/test
#

You won't see any error messages unless you enable logging on autofs. To do that, uncomment the "logging" line on /etc/autofs.conf and set it to "debug". You will then see something like the following when you run "journalctl -u autofs.service":

get_portmap_client: error 0x3 getting portmap client
mount(nfs): no hosts available

You might also want to test an NFSv3 mount and make sure that it's still working as expected. In order to do that, we have to reenable the rpcbind service in the server VM because NSFv3 requires it.

Here are the instructions to test an NFSv3-only mount on the server VM:

Re-enable rpcbind.{socket,service}:

# systemctl unmask rpcbind.socket rpcbind.service
# systemctl start rpcbind.socket rpcbind.service

Edit the nfs-server.service:

# systemctl edit --full nfs-server.service

and change the ExecStart line to look like:

ExecStart=/usr/sbin/rpc.nfsd --no-nfs-version 4

Restart the service:

# systemctl restart nfs-server.service

Here are the instructions to test an NFSv3-only mount on the client VM:

# echo -e 'test\t-fstype=nfs,nfsver=3\tNFS_SERVER_IP:/testshare' > /etc/auto.nfs

(Remember to replace NFS_SERVER_IP with the IP address of the server VM)

Restart autofs:

# systemctl restart autofs.service

Verify that the mount still works:

# ls /mnt/test/
test
#

[ Where problems could occur ]

The patch that fixes the problem has been posted in the upstream mailing list by the maintainer, but hasn't been pushed to the git repository yet (although the maintainer said he would do this soon). I have carefully reviewed the patch and agree with the solution proposed; I have also verified that autofs' dep8 tests are still passing. I find it unlikely that the patch introduces a problem.

If a regression happens, it will most likely manifest during the automount process of NFSv4 shares. We can mitigate the problem by suggesting the use of NFSv3 while we further investigate the issue, and, worst case scenario, we can revert the current fix and work with upstream to improve it.

[ Original Description ]

I'm running a server which provides user home directories as NFSv4 shares. The clients mount these shares via automount + LDAP.

Now I've updated one client from 20.04 to 22.04 (new installation from scratch). Setting up things as before, I cannot get the home directories to mount properly. The LDAP connection does work, and also manual mounting of the NFSv4 shares works. So the problem is in the automount part of the mechanism.
The autofs package version on the 22.04 client is 5.1.8-1ubuntu1.

Setting autofs to verbose mode, I get the following lines in /var/log/syslog on the client.
automount[115692]: mount(nfs): nfs options="nfsvers=4,hard", nobind=0, nosymlink=0, ro=0
automount[115692]: get_nfs_info: called with host <...> proto 6 version 0x40
automount[115692]: get_portmap_client: error 0x3 getting portmap client
automount[115692]: mount(nfs): no hosts available

I think it is the same bug reported on the autofs mailing list here:
https://www.spinics.net/lists/autofs/msg02389.html
Apparently, it has been introduced in the transition of autofs from 5.1.7 to 5.1.8.

A fix has been posted here:
https://www.spinics.net/lists/autofs/msg02391.html

It would be great if this fix could be included in Ubuntu 22.04, such that I can update my clients. As my setup should not be too exotic, I guess there are other people who will face the same problem when they are doing the Ubuntu update to the new LTS version.

Related branches

Revision history for this message
Michael Kiermaier (michael-kiermaier) wrote :

Maybe this one
https://bugs.launchpad.net/ubuntu/+source/autofs/+bug/1965643
is the same bug. It is hard to tell as not much info is given.

description: updated
summary: - autofs fails to mount nfs4 shares
+ autofs fails to mount nfs4 shares with "error 0x3 getting portmap
+ client"
description: updated
description: updated
description: updated
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

First of all, thank you for taking the time to report this bug.

Indeed, the email thread you linked seems to be related to what you're seeing. I have been trying to reproduce the problem locally, but so far I haven't been able to. I'm not using a complex setup like yours; I've just configured a NFS share and setup autofs to automount it, everything in the same VM. Then, I forced nfs-server to use NFSv4 only, edited its systemd service file to not depend on rpcbind, and masked/stopped rpcbind.{service,socket}. Even then I was still able to automount the share successfully.

I'm going to continue investigating tomorrow, but if you have a quick reproducer there I'd appreciate. We will need it in order to proceed with SRU process.

Thanks in advance.

tags: added: server-todo
Changed in autofs (Ubuntu Jammy):
assignee: nobody → Sergio Durigan Junior (sergiodj)
Changed in autofs (Ubuntu Kinetic):
assignee: nobody → Sergio Durigan Junior (sergiodj)
Revision history for this message
Michael Kiermaier (michael-kiermaier) wrote (last edit ):

Many thanks for looking into this.

I guess a "quick reproducer" means that I have to set up 2 fresh 22.04 installs, and then do a minimal configuration of one as a server (provide NFSv4 share) and on the other install as a client (pull in NFSv4 share via autofs mount). This would take some time. I can try to do it, but not right now.

What I did now: In my existing configuration, I bypassed the LDAP mechanism on the client by a direct entry in /etc/auto.master (and commenting out the LDAP ones). Still the same problem.

Also, I had a look into the sources of autofs-5.1.8-1ubuntu1. If I'm not completely mistaken, the patch which I linked above is not present. Therefore, I guess the bug reported (and fixed) on the autofs mailing list is present in Ubuntu 22.04 (as it uses the unpatched autofs 5.1.8), no matter if it is indeed the cause of my problems or not.

So I think it would make sense in any case to either apply the patch posted in the mailing list, or to go back to autofs version 5.1.7.

Revision history for this message
Michael Kiermaier (michael-kiermaier) wrote :

Here is some more info:

The server runs with an enabled UFW firewall. It was set up via

ufw allow from <IP RANGE HIDDEN> to any port nfs proto tcp

Here is the server output of 'ufw status'

Status: active

To Action From
-- ------ ----
OpenSSH ALLOW Anywhere
2049/tcp ALLOW <IP RANGE HIDDEN>
OpenSSH (v6) ALLOW Anywhere (v6)

Now I realized that the original problem disappears if I disable the firewall on the server. I guess the difference is that now the portmapper call gets through.

However, I want my NFSv4 connection to use port 2049 only, as it is working for me with 20.04 (and older) clients.

@Sergio Durigan Junior
By the firewall observation, it makes sense that your first attempt gives a successful automount. I wonder if the mount was cached somehow on the client side when in a second attempt you "forced nfs-server to use NFSv4 only, edited its systemd service file to not depend on rpcbind, and masked/stopped rpcbind".

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thanks for the further info, Michael.

I was able to reproduce the problem here. The trick was to mount the share from another VM. I was able to backport the upstream patch, build autofs with it and verify that it indeed fixes the problem. Unfortunately, it seems that upstream hasn't pushed the patch yet, but I believe it won't be a problem for us.

I will start preparing the MPs for Jammy and Kinetic.

Changed in autofs (Ubuntu Jammy):
status: New → In Progress
Changed in autofs (Ubuntu Kinetic):
status: New → In Progress
Revision history for this message
Michael Kiermaier (michael-kiermaier) wrote :

These are good news, thank you so much!

description: updated
description: updated
Changed in autofs (Ubuntu Jammy):
importance: Undecided → High
Changed in autofs (Ubuntu Kinetic):
importance: Undecided → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package autofs - 5.1.8-1ubuntu2

---------------
autofs (5.1.8-1ubuntu2) kinetic; urgency=medium

  * d/p/fix-nfsv4-only-mounts-should-not-use-rpcbind.patch:
    Make NFSv4-only mounts not depend on rpcbind. (LP: #1970264)

 -- Sergio Durigan Junior <email address hidden> Thu, 28 Apr 2022 23:05:15 -0400

Changed in autofs (Ubuntu Kinetic):
status: In Progress → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Michael, or anyone else affected,

Accepted autofs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autofs/5.1.8-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in autofs (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Michael Kiermaier (michael-kiermaier) wrote :

@Brian Murray
In the process, I see the following:

apt-get install autofs/jammy-proposed
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '5.1.8-1ubuntu1.1' (Ubuntu:22.04/jammy-proposed [amd64]) for 'autofs'
The following packages will be REMOVED:
  autofs-ldap
The following packages will be upgraded:
  autofs

Removing autofs-ldap is not o.k., it seems that some dependency entry needs to be adjusted. (Maybe this should be checked in general: Are there other packages depending on autofs-5.1.8-1ubuntu1?)

Revision history for this message
Michael Kiermaier (michael-kiermaier) wrote :

I can confirm that the package autofs-5.1.8-1ubuntu1.1 solves the original problem, as long as it is a "pure" automount without LDAP involved.

It is clear that a LDAP-based automount cannot work as the autofs-ldap package has been uninstalled.

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote : Re: [Bug 1970264] Re: autofs fails to mount nfs4 shares with "error 0x3 getting portmap client"

On Wednesday, May 04 2022, Michael Kiermaier wrote:

> @Brian Murray
> In the process, I see the following:
>
> apt-get install autofs/jammy-proposed
> Reading package lists... Done
> Building dependency tree... Done
> Reading state information... Done
> Selected version '5.1.8-1ubuntu1.1' (Ubuntu:22.04/jammy-proposed [amd64]) for 'autofs'
> The following packages will be REMOVED:
> autofs-ldap
> The following packages will be upgraded:
> autofs
>
>
> Removing autofs-ldap is not o.k., it seems that some dependency entry
> needs to be adjusted. (Maybe this should be checked in general: Are
> there other packages depending on autofs-5.1.8-1ubuntu1?)

Thanks for verifying, Michael.

Unfortunately I cannot reproduce the problem when using a Jammy
container with both autofs and autofs-ldap installed. Here's what I see
after I enable the -proposed repository and issue the same command
you're using:

# apt-get install autofs/jammy-proposed
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '5.1.8-1ubuntu1.1' (Ubuntu:22.04/jammy-proposed [amd64]) for 'autofs'
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  autofs-ldap
The following packages will be upgraded:
  autofs autofs-ldap
2 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 327 kB of archives.
After this operation, 1024 B of additional disk space will be used.
Do you want to continue? [Y/n]

As you can see, autofs-ldap is marked to be upgraded along autofs, which
makes sense since autofs-ldap has a versioned dependency on autofs.
This leads me to believe that you probably have autofs-ldap pinned or
held by dpkg, which would prevent it from being upgraded. Could you
please double check this?

Thanks,

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

Revision history for this message
Michael Kiermaier (michael-kiermaier) wrote :

Thanks you for the detailed response.

I have to say that this was the first time that I was confronted with the 'proposed'-repository. I followed the instructions at https://wiki.ubuntu.com/Testing/EnableProposed without too much investigation of the precise effect of the single steps.

Thereby, I entered the lines

cat <<EOF >/etc/apt/preferences.d/proposed-updates
# Configure apt to allow selective installs of packages from proposed
Package: *
Pin: release a=$(lsb_release -cs)-proposed
Pin-Priority: 400
EOF

which indeed pinned the autofs-ldap package, as you conjectured.

Now I've added
#apt-get install autofs-ldap/jammy-proposed
and finally everything works as expected.

Many thanks for your help!

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

On Wednesday, May 04 2022, Michael Kiermaier wrote:

> Thanks you for the detailed response.
>
> I have to say that this was the first time that I was confronted with
> the 'proposed'-repository. I followed the instructions at
> https://wiki.ubuntu.com/Testing/EnableProposed without too much
> investigation of the precise effect of the single steps.
>
> Thereby, I entered the lines
>
> cat <<EOF >/etc/apt/preferences.d/proposed-updates
> # Configure apt to allow selective installs of packages from proposed
> Package: *
> Pin: release a=$(lsb_release -cs)-proposed
> Pin-Priority: 400
> EOF
>
> which indeed pinned the autofs-ldap package, as you conjectured.
>
> Now I've added
> #apt-get install autofs-ldap/jammy-proposed
> and finally everything works as expected.
>
> Many thanks for your help!

Awesome! Thank you for confirming. Feel free to adjust the tags in the
bug to reflect the fact that you've verified that the fix works; this
will allow the package to migrate to the -updates repository after the
7-day grace period has passed.

Thanks,

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package autofs - 5.1.8-1ubuntu1.1

---------------
autofs (5.1.8-1ubuntu1.1) jammy; urgency=medium

  * d/p/fix-nfsv4-only-mounts-should-not-use-rpcbind.patch:
    Make NFSv4-only mounts not depend on rpcbind. (LP: #1970264)

 -- Sergio Durigan Junior <email address hidden> Thu, 28 Apr 2022 17:20:37 -0400

Changed in autofs (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for autofs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

tags: removed: server-todo verification-needed
Revision history for this message
Jens Elkner (jelmd) wrote :

automounting NFSv4 shares is still broken in jammy.

It seems to not honor `MOUNT_NFS_DEFAULT_PROTOCOL=4` of /etc/default/autofs anymore. The workaround found so far is to set `OPTIONS="-O vers=4.0"` in the same file (IMHO not a correct fix, but works for now).

Previously (xenial..focal) it worked without OPTION...

Revision history for this message
Jens Elkner (jelmd) wrote :

And yes, setting `[ autofs ] mount_nfs_default_protocol = 4` in /etc/autofs.conf doesn't fix the problem either.

Same thing wrt. `[ NFSMount_Global_Options ] vers=4 nfsvers=4 defaultvers=4` of /etc/nfsmount.conf.

Very confusing all the different files ...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.