gssproxy crashes in libselinux.so.1 on Ubuntu 18.04 when called by rpc.gssd

Can you simplify the configuration needed to reproduce this bug? For example, does it happen when using gssapi authentication with apache, without the NFSv4 bit?

Or can you elaborate a bit more how you have this setup?

And, before I forget, do you have a crash file somewhere in /var/crash?

Status changed to 'Confirmed' because the bug affects multiple users.

Trying to get gssproxy working with NFS (rpc-gssd and rpc-svcgssd) on Ubuntu 20.04
Following https://github.com/gssapi/gssproxy/blob/main/docs/NFS.md

/etc/gssproxy/gssproxy.conf
      [gssproxy]
      debug = true
      debug_level = 3

/etc/gssproxy/25-nfs-server.conf
      [service/nfs-server]
        mechs = krb5
        socket = /run/gssproxy.sock
        cred_store = keytab:/etc/krb5.keytab
        trusted = yes
        kernel_nfsd = yes
        euid = 0

When I start the gssproxy service, either through systemd
or manually with: /usr/sbin/gssproxy --interactive --debug --debug-level=3 --socket=/run/gssproxy.sock

I get this result:
[2021/06/28 14:49:19]: Debug Enabled (level: 3)
[2021/06/28 14:49:19]: Service: nfs-client, Keytab: /etc/krb5.keytab, Enctype: 23
[2021/06/28 14:49:19]: Service: nfs-server, Keytab: /etc/krb5.keytab, Enctype: 23
[2021/06/28 14:49:19]: Client [2021/06/28 14:49:19]: (/usr/sbin/gssproxy) [2021/06/28 14:49:19]: connected (fd = 13)[2021/06/28 14:49:19]: (pid = 7821) (uid = 0) (gid = 0)Segmentation fault (core dumped)

It is the kernel_nfsd = yes config part that causes the segfault
What it does (from the docs linked above)
  ...
  The gssproxy client registers to the kernel by performing 2 actions in the following order:
  * creates a unix socket for kernel communication in /var/run/gssproxy.sock (this path is hardcoded in the kernel and cannot be changed at this time)
  * writes 1 byte in the proc file /proc/net/rpc/use-gss-proxy (the client must be ready to accept a connection from the kernel when this is done, as the kernel we check that the socket is available)
  ...
  It enables the kernel extensions to the protocol (the context is exported as a lucid context for example, and a list of resolved credentials is returned if authentication succeeds)

The proc files seems ok (it was -1 before)
cat /proc/net/rpc/use-gss-proxy
1

The reason we want gssproxy, and not the default rpc-gssd and rpc-svcgssd services is that we are using active directory, and most of our accounts are members of many groups, causing gssd to fail. This is a known issue and is one of the things that gssproxy solves.

>>The reason we did this was to allow the kernel NFS server to handle big tickets like those containing a MS-PAC payload that may be received by a Microsoft client.

I couldn't get it to generate a coredump. But I ran it with valgrind
Hope this helps

valgrind -v /usr/sbin/gssproxy --interactive --debug --debug-level=3 --socket=/run/gssproxy.sock
==29249== Memcheck, a memory error detector
==29249== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==29249== Using Valgrind-3.15.0-608cb11914-20190413 and LibVEX; rerun with -h for copyright info
==29249== Command: /usr/sbin/gssproxy --interactive --debug --debug-level=3 --socket=/run/gssproxy.sock
==29249==
--29249-- Valgrind options:
--29249-- -v
--29249-- Contents of /proc/version:
--29249-- Linux version 5.4.0-74-generic (buildd@lgw01-amd64-038) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #83-Ubuntu SMP Sat May 8 02:35:39 UTC 2021
--29249--
--29249-- Arch and hwcaps: AMD64, LittleEndian, amd64-cx16-lzcnt-rdtscp-sse3-ssse3-avx-avx2-rdrand
--29249-- Page sizes: currently 4096, max supported 4096
--29249-- Valgrind library directory: /usr/lib/x86_64-linux-gnu/valgrind
--29249-- Reading syms from /usr/sbin/gssproxy
--29249-- object doesn't have a symbol table
--29249-- Reading syms from /usr/lib/x86_64-linux-gnu/ld-2.31.so
--29249-- Considering /usr/lib/x86_64-linux-gnu/ld-2.31.so ..
--29249-- .. CRC mismatch (computed 975d0390 wanted 30bd717f)
--29249-- Considering /lib/x86_64-linux-gnu/ld-2.31.so ..
--29249-- .. CRC mismatch (computed 975d0390 wanted 30bd717f)
--29249-- Considering /usr/lib/debug/lib/x86_64-linux-gnu/ld-2.31.so ..
--29249-- .. CRC is valid
--29249-- Reading syms from /usr/lib/x86_64-linux-gnu/valgrind/memcheck-amd64-linux
--29249-- object doesn't have a symbol table
--29249-- object doesn't have a dynamic symbol table
--29249-- Scheduler: using generic scheduler lock implementation.
--29249-- Reading suppressions file: /usr/lib/x86_64-linux-gnu/valgrind/default.supp
==29249== embedded gdbserver: reading from /tmp/vgdb-pipe-from-vgdb-to-29249-by-root-on-???
==29249== embedded gdbserver: writing to /tmp/vgdb-pipe-to-vgdb-from-29249-by-root-on-???
==29249== embedded gdbserver: shared mem /tmp/vgdb-pipe-shared-mem-vgdb-29249-by-root-on-???
==29249==
==29249== TO CONTROL THIS PROCESS USING vgdb (which you probably
==29249== don't want to do, unless you know exactly what you're doing,
==29249== or are doing some strange experiment):
==29249== /usr/lib/x86_64-linux-gnu/valgrind/../../bin/vgdb --pid=29249 ...command...
==29249==
==29249== TO DEBUG THIS PROCESS USING GDB: start GDB like this
==29249== /path/to/gdb /usr/sbin/gssproxy
==29249== and then give GDB the following command
==29249== target remote | /usr/lib/x86_64-linux-gnu/valgrind/../../bin/vgdb --pid=29249
==29249== --pid is optional if only one valgrind process is running
==29249==
--29249-- REDIR: 0x4022e10 (ld-linux-x86-64.so.2:strlen) redirected to 0x580c9ce2 (???)
--29249-- REDIR: 0x4022be0 (ld-linux-x86-64.so.2:index) redirected to 0x580c9cfc (???)
--29249-- Reading syms from /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_core-amd64-linux.so
--29249-- object doesn't have a symbol table
--29249-- Reading syms from /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so
--29249-- object doesn't have a symbol t...

gssproxy/focal,now 0.8.2-2 amd64 [installed]
libselinux1/focal,now 3.0-1build2 amd64 [installed,automatic]

I am taking care of this one.

It is important to mention that this could arguably be considered to be a bug on libselinux, given that it shouldn't dereference pointers without checking first (especially pointers that were passed to the library by its clients). However, in this case it makes sense to fix this in gssproxy as well.

The upstream bug is here: https://pagure.io/gssproxy/issue/256

The patch is here: https://github.com/gssapi/gssproxy/commit/3b77666d463105fc485c0f269feaf0ed1061a769

I removed the krb5 component because the bug is not in it.

I can confirm that manual build and install of gssproxy 0.8.4 works on my ubuntu 20.04 server.
(that version has the patch mentioned above)

gssproxy solves my original issue of rpc-svcgssd hanging on large kerberos tickets https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1466654

Hopefully this patch find its way fast through the official ubuntu release channel

Hello Robert, or anyone else affected,

Accepted gssproxy into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gssproxy/0.8.2-2ubuntu0.21.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Robert, or anyone else affected,

Accepted gssproxy into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gssproxy/0.8.2-2ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

The fix for this bug has been awaiting testing feedback in the -proposed repository for focal for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

Just tested the proposed gssproxy fix, and can confirm that it solved the issue
Tested on Ubuntu Focal (20.04)

Performing the verification for Focal.

First, install the problematic package and reproduce the error:

# apt policy gssproxy
gssproxy:
  Installed: 0.8.2-2
  Candidate: 0.8.2-2
  Version table:
 *** 0.8.2-2 500
        500 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages
        100 /var/lib/dpkg/status
# /usr/sbin/gssproxy --interactive --debug --debug-level=3 --socket=/run/gssproxy.sock
[2021/10/04 18:30:15]: Debug Enabled (level: 3)
[2021/10/04 18:30:15]: Keytab /etc/krb5.keytab has no content (-1765328203)
[2021/10/04 18:30:15]: Service: nfs-server, Enckey: [ephemeral], Enctype: 18
[2021/10/04 18:30:15]: Client [2021/10/04 18:30:15]: (/usr/sbin/gssproxy) [2021/10/04 18:30:15]: connected (fd = 12)[2021/10/04 18:30:15]: (pid = 1877) (uid = 0) (gid = 0)Segmentation fault (core dumped)

Now, install the version from -proposed and verify that it works correctly:

# apt install gssproxy
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  gssproxy
1 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.
Need to get 88.4 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-proposed/universe amd64 gssproxy amd64 0.8.2-2ubuntu0.20.04.1 [88.4 kB]
Fetched 88.4 kB in 0s (224 kB/s)
(Reading database ... 15068 files and directories currently installed.)
Preparing to unpack .../gssproxy_0.8.2-2ubuntu0.20.04.1_amd64.deb ...
Unpacking gssproxy (0.8.2-2ubuntu0.20.04.1) over (0.8.2-2) ...
Setting up gssproxy (0.8.2-2ubuntu0.20.04.1) ...
# apt policy gssproxy
gssproxy:
  Installed: 0.8.2-2ubuntu0.20.04.1
  Candidate: 0.8.2-2ubuntu0.20.04.1
  Version table:
 *** 0.8.2-2ubuntu0.20.04.1 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     0.8.2-2 500
        500 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages
# /usr/sbin/gssproxy --interactive --debug --debug-level=3 --socket=/run/gssproxy.sock
[2021/10/04 18:31:59]: Debug Enabled (level: 3)
[2021/10/04 18:31:59]: Keytab /etc/krb5.keytab has no content (-1765328203)
[2021/10/04 18:31:59]: Service: nfs-server, Enckey: [ephemeral], Enctype: 18
[2021/10/04 18:31:59]: Client [2021/10/04 18:31:59]: (/usr/sbin/gssproxy) [2021/10/04 18:31:59]: connected (fd = 12)[2021/10/04 18:31:59]: (pid = 2244) (uid = 0) (gid = 0)[2021/10/04 18:31:59]:
^C[2021/10/04 18:32:10]: Exiting after receiving a signal

Therefore, the new package indeed fixes the problem.

Performing the verification for Hirsute.

First, install the problematic package and reproduce the error:

# apt policy gssproxy
gssproxy:
  Installed: 0.8.2-2
  Candidate: 0.8.2-2
  Version table:
 *** 0.8.2-2 500
        500 http://archive.ubuntu.com/ubuntu hirsute/universe amd64 Packages
        100 /var/lib/dpkg/status
# /usr/sbin/gssproxy --interactive --debug --debug-level=3 --socket=/run/gssproxy.sock
[2021/10/04 18:34:27]: Debug Enabled (level: 3)
[2021/10/04 18:34:27]: Keytab /etc/krb5.keytab has no content (-1765328203)
[2021/10/04 18:34:27]: Service: nfs-server, Enckey: [ephemeral], Enctype: 18
[2021/10/04 18:34:27]: Client [2021/10/04 18:34:27]: (/usr/sbin/gssproxy) [2021/10/04 18:34:27]: connected (fd = 12)[2021/10/04 18:34:27]: (pid = 1676) (uid = 0) (gid = 0)Segmentation fault (core dumped)

Now, install the version from -proposed and verify that it works correctly:

# apt install gssproxy
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
  gssproxy
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 88.0 kB of archives.
After this operation, 13.3 kB disk space will be freed.
Get:1 http://archive.ubuntu.com/ubuntu hirsute-proposed/universe amd64 gssproxy amd64 0.8.2-2ubuntu0.21.04.1 [88.0 kB]
Fetched 88.0 kB in 0s (236 kB/s)
(Reading database ... 15225 files and directories currently installed.)
Preparing to unpack .../gssproxy_0.8.2-2ubuntu0.21.04.1_amd64.deb ...
Unpacking gssproxy (0.8.2-2ubuntu0.21.04.1) over (0.8.2-2) ...
Setting up gssproxy (0.8.2-2ubuntu0.21.04.1) ...
# apt policy gssproxy
gssproxy:
  Installed: 0.8.2-2ubuntu0.21.04.1
  Candidate: 0.8.2-2ubuntu0.21.04.1
  Version table:
 *** 0.8.2-2ubuntu0.21.04.1 500
        500 http://archive.ubuntu.com/ubuntu hirsute-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     0.8.2-2 500
        500 http://archive.ubuntu.com/ubuntu hirsute/universe amd64 Packages
# /usr/sbin/gssproxy --interactive --debug --debug-level=3 --socket=/run/gssproxy.sock
[2021/10/04 18:35:20]: Debug Enabled (level: 3)
[2021/10/04 18:35:20]: Keytab /etc/krb5.keytab has no content (-1765328203)
[2021/10/04 18:35:20]: Service: nfs-server, Enckey: [ephemeral], Enctype: 18
[2021/10/04 18:35:20]: Client [2021/10/04 18:35:20]: (/usr/sbin/gssproxy) [2021/10/04 18:35:20]: connected (fd = 12)[2021/10/04 18:35:20]: (pid = 2029) (uid = 0) (gid = 0)[2021/10/04 18:35:20]:
^C[2021/10/04 18:35:24]: Exiting after receiving a signal

Therefore, the new package indeed fixes the problem.

This bug was fixed in the package gssproxy - 0.8.2-2ubuntu0.21.04.1

---------------
gssproxy (0.8.2-2ubuntu0.21.04.1) hirsute; urgency=medium

  * d/p/0001-Fix-handling-of-selinux-context-when-NULL.patch:
    Fix handling of SELinux context when NULL. Do not segfault
    when gssproxy is called by rpc.gssd. (LP: #1788459)

 -- Sergio Durigan Junior <email address hidden> Wed, 30 Jun 2021 14:44:13 -0400

The verification of the Stable Release Update for gssproxy has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package gssproxy - 0.8.2-2ubuntu0.20.04.1

---------------
gssproxy (0.8.2-2ubuntu0.20.04.1) focal; urgency=medium

  * d/p/0001-Fix-handling-of-selinux-context-when-NULL.patch:
    Fix handling of SELinux context when NULL. Do not segfault
    when gssproxy is called by rpc.gssd. (LP: #1788459)

 -- Sergio Durigan Junior <email address hidden> Wed, 30 Jun 2021 13:33:56 -0400

This bug doesn't apply to Impish.