freeradius with freeradius-python3 fails to start out of the box

Bug #1873923 reported by synapse
38
This bug affects 6 people
Affects Status Importance Assigned to Milestone
freeradius (Ubuntu)
Fix Released
Medium
Andreas Hasenack
Focal
Fix Released
Medium
Andreas Hasenack

Bug Description

[Impact]
The python3 rlm module cannot be loaded.

[Test Plan]

# pull the necessary attachments
wget https://bugs.launchpad.net/ubuntu/+source/freeradius/+bug/1873923/+attachment/5580525/+files/ubuntu_example.py https://bugs.launchpad.net/ubuntu/+source/freeradius/+bug/1873923/+attachment/5580526/+files/python3

# install freeradius-python3
sudo apt update
sudo apt install freeradius-python3

# enable python3 for auth (this command is one line)
sudo sed -i -r '/^#[[:blank:]]*auth_log/a python3' /etc/freeradius/3.0/sites-available/default

# copy python example
sudo cp ubuntu_example.py /etc/freeradius/3.0/mods-config/python3/

# copy python3 config
sudo cp python3 /etc/freeradius/3.0/mods-available/python3

# enable the python3 module
sudo ln -s /etc/freeradius/3.0/mods-available/python3 /etc/freeradius/3.0/mods-enabled/

# restart freeradius. Restart will fail without the fixed package installed
sudo systemctl restart freeradius

Error in journal when the fix is not installed:
Apr 13 20:54:03 f-freeradius-1873923 freeradius[5998]: Python version: 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
Apr 13 20:54:03 f-freeradius-1873923 freeradius[5998]: Libpython is not found among linked libraries
Apr 13 20:54:03 f-freeradius-1873923 freeradius[5998]: Failed loading libpython symbols into global symbol table
Apr 13 20:54:03 f-freeradius-1873923 freeradius[5998]: *** example.py instantiate ***

and

Apr 13 20:55:31 f-freeradius-1873923 freeradius[6165]: *** example.py detach ***
Apr 13 20:55:31 f-freeradius-1873923 systemd[1]: freeradius.service: Control process exited, code=dumped, status=11/SEGV

# Install the fixed packages from proposed
# Test authentication with the python3 script we installed.
# This script will accept the ubuntu user with any password, and reject
# any other user, regardless of the password

$ radtest ubuntu anypass 127.0.0.1 0 testing123
Sent Access-Request Id 59 from 0.0.0.0:33632 to 127.0.0.1:1812 length 76
        User-Name = "ubuntu"
        User-Password = "anypass"
        NAS-IP-Address = 10.0.100.53
        NAS-Port = 0
        Message-Authenticator = 0x00
        Cleartext-Password = "anypass"
Received Access-Accept Id 59 from 127.0.0.1:1812 to 127.0.0.1:33632 length 35
        Reply-Message = "Hello ubuntu!"

$ radtest anotheruser neverworks 127.0.0.1 0 testing123
Sent Access-Request Id 14 from 0.0.0.0:34461 to 127.0.0.1:1812 length 81
        User-Name = "anotheruser"
        User-Password = "neverworks"
        NAS-IP-Address = 10.0.100.53
        NAS-Port = 0
        Message-Authenticator = 0x00
        Cleartext-Password = "neverworks"
Received Access-Reject Id 14 from 127.0.0.1:1812 to 127.0.0.1:34461 length 41
        Reply-Message = "You are not ubuntu!"
(0) -: Expected Access-Accept got Access-Reject

[Where problems could occur]
I don't know how complex the python3 scripts run by rlm_python3 can be. Maybe if there are complex imports or other interactions, it could show this fix to be incomplete, as there were other upstream fixes for rlm_python3 that landed in 3.0.21 and later. I checked the 3.0.21 ones (https://github.com/FreeRADIUS/freeradius-server/issues/3193#issuecomment-564142712), and believe I have the correct set of minimal fixes. In any case, this upload is an improvement already, because before one could not even load rlm_python3. And if you tried to use it really, it would segfault.

That being said, freeradius is a gatekeeper. Authentication and authorization is a big deal, and if this update introduces a bug, it could manifest itself either by allowing something it shouldn't, or not allowing anything. Given the patches, this will be restricted to the python3 module, which couldn't be loaded anyway.

These modules deep inside src/modules have their own autoconf scripts, and they don't get regenerated by the build process from the respective source files (configure.ac and others). At this time I didn't have to patch them, but, as can be seen in the jammy upload, I had to patch configure directly (and not configure.ac) because of fear of introducing a regression, since those deep configure scripts are NOT regenerated by the debian packaging builds: just the top level one is regenerated, at the root of the source tree, and who knows when was the last time all those configure scripts were generated. And jammy has an updated autoconf (2.71), which broke other packages. In any case, this concern does not apply to this particular focal update, but might in the future.

[Other Info]
This SRU has two patches. One is the actual fix, which fixes the name of the python library that freeradius attempts to load. The other fix is to not try to instantiate the python3 module if the instantiate and detach methods are not defined, which would trigger a failure with the simple reproducer from comment #1.

I added a DEP8 test to the focal package that doesn't exist in any other release at this time. As soon as Ubuntu K opens up, I'll update the freeradius package there as well with this test (see https://bugs.launchpad.net/ubuntu/+source/freeradius/+bug/1969381)

UPDATE: PR to update the DEP8 test in kinetic's freeradius package: https://code.launchpad.net/~ahasenack/ubuntu/+source/freeradius/+git/freeradius/+merge/421436

[Original Description]

I created a git repo with the reproduction: https://github.com/gergelykalman/freeradius-python3-bug

Package version:
Version: 3.0.20+dfsg-3build1

The error message:
-----------------------
Mon Apr 20 17:12:57 2020 : Debug: rlm_passwd: nfields: 3 keyfield 0(User-Name) listable: no
Mon Apr 20 17:12:57 2020 : Debug: # Instantiating module "python3" from file /etc/freeradius/3.0/mods-enabled/python3
Mon Apr 20 17:12:57 2020 : Info: Python version: 3.8.2 (default, Mar 13 2020, 10:14:16) [GCC 9.3.0]
Mon Apr 20 17:12:57 2020 : Warning: Libpython is not found among linked libraries
Mon Apr 20 17:12:57 2020 : Warning: Failed loading libpython symbols into global symbol table
Mon Apr 20 17:12:57 2020 : Error: do_python_single:568, instantiate - pRet is NULL
Mon Apr 20 17:12:57 2020 : Error: python_error_log:200, Exception type: <class 'SystemError'>, Exception value: null argument to internal routine
Mon Apr 20 17:12:57 2020 : Error: do_python_single:676, instantiate - RLM_MODULE_FAIL
-----------------------

Related branches

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Thanks for the bug report! I have tried to reproduce this issue in a LXD Focal container and it fails while loading the python3 module as mentioned above:

$ apt-get install -y freeradius freeradius-python3
$ ln -s /etc/freeradius/3.0/mods-available/python3 /etc/freeradius/3.0/mods-enabled/
$ systemctl stop freeradius
$ freeradius -Xx
...
Wed Apr 22 18:39:51 2020 : Debug: # Instantiating module "python3" from file /etc/freeradius/3.0/mods-enabled/python3
Wed Apr 22 18:39:51 2020 : Info: Python version: 3.8.2 (default, Mar 13 2020, 10:14:16) [GCC 9.3.0]
Wed Apr 22 18:39:51 2020 : Warning: Libpython is not found among linked libraries
Wed Apr 22 18:39:51 2020 : Warning: Failed loading libpython symbols into global symbol table
Wed Apr 22 18:39:52 2020 : Error: do_python_single:568, instantiate - pRet is NULL
Wed Apr 22 18:39:52 2020 : Error: python_error_log:200, Exception type: <class 'SystemError'>, Exception value: null argument to internal routine
Wed Apr 22 18:39:52 2020 : Error: do_python_single:676, instantiate - RLM_MODULE_FAIL
...

Changed in freeradius (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Daniel Kucera (danmaneu) wrote :

Is someone working on this?

Revision history for this message
Daniel Kucera (danmaneu) wrote :

I've upgraded from buster where python(2) module was working. Now in focal it seems none of python or python3 works.

Python2 gives:
/etc/freeradius/3.0/mods-enabled/python[9]: Failed to link to module 'rlm_python': /usr/lib/freeradius/rlm_python.so: cannot open shared object file: No such file or directory

Revision history for this message
Athos Ribeiro (athos-ribeiro) wrote :

It would be nice to report this to Debian as well. Then, we could get this one fixed in Debian and keep it as a sync.

Changed in freeradius (Ubuntu):
status: Confirmed → Triaged
tags: added: server-todo
Changed in freeradius (Ubuntu):
assignee: nobody → Lena Voytek (lvoytek)
Lena Voytek (lvoytek)
Changed in freeradius (Ubuntu Focal):
status: New → Confirmed
assignee: nobody → Lena Voytek (lvoytek)
Lena Voytek (lvoytek)
Changed in freeradius (Ubuntu Focal):
status: Confirmed → In Progress
Revision history for this message
Rolf Leggewie (r0lf) wrote :

other issues with the package that prevent it to start in jammy:

- directory sites-enabled is not created
- "The server is not configured to listen on any ports. Cannot start"

Changed in freeradius (Ubuntu Focal):
importance: Undecided → Medium
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

A similar error happens with the current version in jammy:

Wed Mar 23 16:58:54 2022 : Error: /etc/freeradius/3.0/mods-enabled/python3[9]: Failed to link to module 'rlm_python3': /usr/lib/freeradius/rlm_python3.so: undefined symbol: PyTuple_Type

Looks like it's https://bugzilla.redhat.com/show_bug.cgi?id=1917157:
# ldd /usr/lib/freeradius/rlm_python3.so
        linux-vdso.so.1 (0x00007ffe65fe1000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb4037c9000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb403a03000)

Changed in freeradius (Ubuntu):
assignee: Lena Voytek (lvoytek) → Andreas Hasenack (ahasenack)
Revision history for this message
Andreas Hasenack (ahasenack) wrote (last edit ):

On debian sid (3.0.25+dfsg-1+b1), where it starts:

# ldd /usr/lib/freeradius/rlm_python3.so
 linux-vdso.so.1 (0x00007ffef8ba8000)
 libpython3.9.so.1.0 => /lib/x86_64-linux-gnu/libpython3.9.so.1.0 (0x00007fe1e1111000)
 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe1e10f0000)
 libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe1e10ea000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe1e0f11000)
 /lib64/ld-linux-x86-64.so.2 (0x00007fe1e16b3000)
 libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fe1e0ee0000)
 libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fe1e0ec3000)
 libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fe1e0ebc000)
 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe1e0d78000)

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I wonder if this version comparison started failing with python 3.10...

AX_COMPARE_VERSION(${PYTHON_VERSION}, [ge], [3.8], [EMBED="--embed"], [])

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Bah, yeah, 3.10 is converted into 3.1, which is lower than 3.8 :D

checking for python3-config... python3-config
configure: python3-config's cflags were "-I/usr/include/python3.10 -I/usr/include/python3.10 -Wno-unused-result -Wsign-compare -g -fstack-protector-strong -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O2 -Wall"
configure: Sanitized cflags were " -isystem/usr/include/python3.10 -isystem/usr/include/python3.10 -fstack-protector-strong -fwrapv "
checking for gawk... gawk
++ echo 3.1
++ sed -e 's/\([0-9]*\)/Z\1Z/g' -e 's/Z\([0-9]\)Z/Z0\1Z/g' -e 's/Z\([0-9][0-9]\)Z/Z0\1Z/g' -e 's/Z\([0-9][0-9][0-9]\)Z/Z0\1Z/g' -e 's/[^0-9]//g'
+ ax_compare_version_A=00030001
++ echo 3.8
++ sed -e 's/\([0-9]*\)/Z\1Z/g' -e 's/Z\([0-9]\)Z/Z0\1Z/g' -e 's/Z\([0-9][0-9]\)Z/Z0\1Z/g' -e 's/Z\([0-9][0-9][0-9]\)Z/Z0\1Z/g' -e 's/[^0-9]//g'
+ ax_compare_version_B=00030008

Changed in freeradius (Ubuntu):
status: Triaged → In Progress
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

The AX_COMPARE_VERSION function it fine, what's wrong is the python version it is fed: 3.1

That comes from "import sys; sys.stdout.write(sys.version[:3])" which doesn't work for minor versions with more than one digit...

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package freeradius - 3.0.26~dfsg~git20220223.1.00ed0241fa-0ubuntu3

---------------
freeradius (3.0.26~dfsg~git20220223.1.00ed0241fa-0ubuntu3) jammy; urgency=medium

  * d/p/fix-python-version-parsing.patch: fix parsing of python versions
    with multiple digits (LP: #1873923)

 -- Andreas Hasenack <email address hidden> Wed, 23 Mar 2022 20:13:55 -0300

Changed in freeradius (Ubuntu):
status: In Progress → Fix Released
Changed in freeradius (Ubuntu Focal):
assignee: Lena Voytek (lvoytek) → Andreas Hasenack (ahasenack)
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I found some issues and am working through them in https://code.launchpad.net/~ahasenack/ubuntu/+source/freeradius/+git/freeradius/+ref/focal-freeradius-py3.8-1873923. It's not ready yet, though.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I identified the necessary fixes and came up with a test case.

description: updated
description: updated
description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :
description: updated
description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :
description: updated
description: updated
description: updated
description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Focal update uploaded, it's in focal-unapproved now

$ dput ubuntu ../freeradius_3.0.20+dfsg-3ubuntu0.1_source.changes
Uploading freeradius using ftp to ubuntu (host: upload.ubuntu.com; directory: /ubuntu)
running badauthor: Stop if uploading with root@ or ubuntu@ email adresses.
running supported-distribution: check whether the target distribution is currently supported (using distro-info)
{'allowed': ['release', 'proposed', 'backports', 'security'], 'known': ['release', 'proposed', 'updates', 'backports', 'security']}
running gpg: check GnuPG signatures before the upload
running updatemaintainer: Stop if ubuntu changes are without ubuntu maintainer.
running check-debs: makes sure the upload contains a binary package
running releasemismatch: Warn about mismatching suffixesg e.g. focal with a XX.YY not being 20.04
running ppaforppaonly: Stop uploads to the archive with or to ppa without ~ppa suffix.
running required-fields: check whether a field is present and non-empty in the changes file
running placeholderbug: Stop if using common placeholder numbers as bug reference.
running gitubuntu: Warn if uploading without git-ubuntu Vcs-* entries.
running checksum: verify checksums before uploading
running nobug: Stop if uploading without any bug reference.
running suite-mismatch: check the target distribution for common errors
Uploading freeradius_3.0.20+dfsg-3ubuntu0.1.dsc
Uploading freeradius_3.0.20+dfsg-3ubuntu0.1.debian.tar.xz
Uploading freeradius_3.0.20+dfsg-3ubuntu0.1_source.buildinfo
Uploading freeradius_3.0.20+dfsg-3ubuntu0.1_source.changes

description: updated
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello synapse, or anyone else affected,

Accepted freeradius into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/freeradius/3.0.20+dfsg-3ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in freeradius (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Focal verification

Problem reproduced with focal packages 3.0.20+dfsg-3build1

May 12 19:05:28 focal-freeradius-1873923 freeradius[3976]: Python version: 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
May 12 19:05:28 focal-freeradius-1873923 freeradius[3976]: Libpython is not found among linked libraries
May 12 19:05:28 focal-freeradius-1873923 freeradius[3976]: Failed loading libpython symbols into global symbol table
May 12 19:05:28 focal-freeradius-1873923 freeradius[3976]: *** example.py instantiate ***
May 12 19:05:28 focal-freeradius-1873923 freeradius[3976]: rlm_cache (cache_eap): Driver rlm_cache_rbtree (module rlm_cache_rbtree) loaded and linked
May 12 19:05:28 focal-freeradius-1873923 freeradius[3976]: Ignoring "sql" (see raddb/mods-available/README.rst)
May 12 19:05:28 focal-freeradius-1873923 freeradius[3976]: Ignoring "ldap" (see raddb/mods-available/README.rst)
May 12 19:05:28 focal-freeradius-1873923 freeradius[3976]: # Skipping contents of 'if' as it is always 'false' -- /etc/freeradius/3.0/sites-enabled/inner-tunnel:336
May 12 19:05:28 focal-freeradius-1873923 freeradius[3976]: radiusd: #### Skipping IP addresses and Ports ####
May 12 19:05:28 focal-freeradius-1873923 freeradius[3976]: Configuration appears to be OK
May 12 19:05:28 focal-freeradius-1873923 freeradius[3976]: *** example.py detach ***
May 12 19:05:28 focal-freeradius-1873923 systemd[1]: freeradius.service: Control process exited, code=dumped, status=11/SEGV

Installing updated packages from proposed:
ubuntu@focal-freeradius-1873923:~$ apt-cache policy freeradius-python3
freeradius-python3:
  Installed: 3.0.20+dfsg-3ubuntu0.1
  Candidate: 3.0.20+dfsg-3ubuntu0.1
  Version table:
 *** 3.0.20+dfsg-3ubuntu0.1 500
        500 http://br.archive.ubuntu.com/ubuntu focal-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     3.0.20+dfsg-3build1 500
        500 http://br.archive.ubuntu.com/ubuntu focal/universe amd64 Packages

This time, service restarts fine:
ubuntu@focal-freeradius-1873923:~$ sudo systemctl restart freeradius
ubuntu@focal-freeradius-1873923:~$

ubuntu user test passes:
ubuntu@focal-freeradius-1873923:~$ radtest ubuntu anypass 127.0.0.1 0 testing123
Sent Access-Request Id 220 from 0.0.0.0:54648 to 127.0.0.1:1812 length 76
 User-Name = "ubuntu"
 User-Password = "anypass"
 NAS-IP-Address = 10.0.100.10
 NAS-Port = 0
 Message-Authenticator = 0x00
 Cleartext-Password = "anypass"
Received Access-Accept Id 220 from 127.0.0.1:1812 to 127.0.0.1:54648 length 35
 Reply-Message = "Hello ubuntu!"

A non-ubuntu user is rejected as expected:
ubuntu@focal-freeradius-1873923:~$ radtest anotheruser neverworks 127.0.0.1 0 testing123
Sent Access-Request Id 150 from 0.0.0.0:57202 to 127.0.0.1:1812 length 81
 User-Name = "anotheruser"
 User-Password = "neverworks"
 NAS-IP-Address = 10.0.100.10
 NAS-Port = 0
 Message-Authenticator = 0x00
 Cleartext-Password = "neverworks"
Received Access-Reject Id 150 from 127.0.0.1:1812 to 127.0.0.1:57202 length 41
 Reply-Message = "You are not ubuntu!"
(0) -: Expected Access-Accept got Access-Reject

Focal verification succeeded.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for freeradius has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package freeradius - 3.0.20+dfsg-3ubuntu0.1

---------------
freeradius (3.0.20+dfsg-3ubuntu0.1) focal; urgency=medium

  * Fix loading of rlm_python3 (LP: #1873923):
    - d/p/py3.8-libname-fix.patch: fix the python library name that
      freeradius is looking for when loading the rlm_python3 module
    - d/p/dont_call_undeclared.patch: don't attempt to load the python
      module if "instantiate" and "detach" are not declared
    - Add test for rlm_python3
      + d/t/control: new rlm_python3 test
      + d/t/rlm_python3-test: test the rlm_python3 module
      + d/t/rlm_python3-data/*: test files
  * d/t/test-freeradius.py: test more authentication mechanisms
    (cherry-picked from Ubuntu 22.04)
  * d/t/freeradius: run python tests in verbose mode

 -- Andreas Hasenack <email address hidden> Thu, 14 Apr 2022 17:32:00 -0300

Changed in freeradius (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.