snmpd crashes with segfault (libnetsnmpmibs.so.15.1.2)

rhhoek I've marked this bug private, as it includes your snmpd community name, which you may not have wanted to be publicly available. If not, then I suggest uploading new versions and redacting the community name before marking the report public again.

I'm also marking this as Confirmed. I've just tested this on a lucid amd64 chroot, and something in your snmpd.conf causes the segfault. I was able to get a core dump by starting the service with your snmpd.conf after setting ulimit -c unlimited

Also setting Importance to high.

Hi Clint,

Thanks for your concern about the snmp community name. In the original uploaded config the community name was a fake one already (zwaargeheim = verysecret). Now I have changed the IP-address too.
It is no problem to me to change the visibility back to public.
Good to see the the bug is reproducible.

I am seeing the same segfault on my lucid boxes, config file attached with sensitive information removed. None of the extended checks are actually being done on the machines that are crashing, as the first hosts we upgraded to lucid do not do anything that complex -- run lighttpd and that is all. They do push 200Mbps of traffic fairly constantly, but load is always below 1.0.

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please try to obtain a backtrace following the instructions at http://wiki.ubuntu.com/DebuggingProgramCrash and upload the backtrace (as an attachment) to the bug report. This will greatly help us in tracking down your problem.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2011-03-07 15:01, Chuck Short wrote:
> Thank you for taking the time to report this bug and helping to make
> Ubuntu better. Please try to obtain a backtrace following the
> instructions at http://wiki.ubuntu.com/DebuggingProgramCrash and upload

I have followed the instructions. Until now, snmpd has crashed only
ones. Do I have to wait for a crash again before sending in an backtrace?

> the backtrace (as an attachment) to the bug report. This will greatly
> help us in tracking down your problem.
>

- --

Met vriendelijke groeten,

Roel Hoek
ICT Service Centre
University of Twente, P.O.Box 217, 7500 AE Enschede, The Netherlands
Telephone +31 53 489 4598, Fax +31 53 489 2383
<email address hidden>; http://www.utwente.nl/icts
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk12GtUACgkQJwlRSGnYBcY1RgCfT06hvgFfGDFniywLKXDZ1oTV
a6YAniRS9k0xsvjPweo1LyaHexh4LIiL
=roZM
-----END PGP SIGNATURE-----

Crashes on ubuntu 20.04 too:
---

...
Aug 19 18:30:07 server snmpd[1308748]: Cannot statfs /run/docker/netns/cf01dc8e9bbc: Permission denied
Aug 19 18:30:16 server snmpd[1308748]: error on subcontainer 'ifTable container' remove (-1)
Aug 19 18:30:19 server kernel: snmpd[1308748]: segfault at 91 ip 00007f6d89fde775 sp 00007fff92349a00 error 4 in libnetsnmpmibs.so.35.0.0[7f6d89f44000+d3000]

@Christoph could you please try to provide a backtrace to us? Also if you have any settings in your config file different from the default, could you share it with us? You are using a newer version than the one originally reported in this bug, so that would be great to determine if this is a similar bug or something else.

Here you can find some information about debugging:

https://wiki.ubuntu.com/DebuggingProgramCrash

Hi I'm seeing these on one of the servers running docker. 20.04 running snmpd-5.8+dfsg-2ubuntu2.3:

kernel: snmpd[1866629]: segfault at 6f ip 00007fa535732775 sp 00007fffaf58c470 error 4 in libnetsnmpmibs.so.35.0.0[7fa535698000+d3000]

I was able to get some details, probably not enough from the crash data:

#0 __GI___libc_free (mem=0x2) at malloc.c:3102
        ar_ptr = <optimized out>
        p = <optimized out>
        hook = 0x0
#1 0x00007f4170726f6b in netsnmp_access_interface_entry_free (entry=0x55cbcc574880) at if-mib/data_access/interface.c:320
        __func__ = "netsnmp_access_interface_entry_free"
#2 0x00007f41706fc718 in ifTable_rowreq_ctx_cleanup (rowreq_ctx=rowreq_ctx@entry=0x55cbcc587fe0) at if-mib/ifTable/ifTable.c:228
        __func__ = "ifTable_rowreq_ctx_cleanup"
#3 0x00007f417072ae22 in ifTable_release_rowreq_ctx (rowreq_ctx=0x55cbcc587fe0) at if-mib/ifTable/ifTable_interface.c:626
        __func__ = "ifTable_release_rowreq_ctx"
#4 0x00007f4170578848 in _ssll_for_each (c=<optimized out>, f=0x7f417072bbd0 <_delete_missing_interface>, context=0x55cbcc3e7c20) at container_list_ssll.c:284
        sl = <optimized out>
        curr = 0x55cbcc594a50
#5 0x00007f417072cef8 in ifTable_container_load (container=0x55cbcc3e7c20) at if-mib/ifTable/ifTable_data_access.c:643
        cdc = {current = 0x55cbcc59b720, deleted = 0x55cbcc59c910}
        __func__ = "ifTable_container_load"
#6 0x00007f41709074ad in _cache_load (cache=0x55cbcc3e7b90) at helpers/cache_handler.c:735
        ret = -1
#7 0x00007f41705524b7 in run_alarms () at snmp_alarm.c:218
        a = 0x55cbcc3e9820
        clientreg = 5
        t_now = {tv_sec = 1922816, tv_usec = 13682}
        __func__ = "run_alarms"
#8 0x000055cbcba3a864 in receive () at snmpd.c:1376
        numfds = 8
        readfds = {lfs_setsize = 1024, lfs_setptr = 0x7fff01800d50, lfs_set = {fds_bits = {0 <repeats 16 times>}}}
        writefds = {lfs_setsize = 1024, lfs_setptr = 0x7fff01800de0, lfs_set = {fds_bits = {0 <repeats 16 times>}}}
        exceptfds = {lfs_setsize = 1024, lfs_setptr = 0x7fff01800e70, lfs_set = {fds_bits = {0 <repeats 16 times>}}}
        timeout = {tv_sec = 0, tv_usec = 0}
        tvp = <optimized out>
        count = 0
        block = 0
        i = <optimized out>
        sd = <optimized out>
        __func__ = "receive"
#9 0x000055cbcba3a10d in main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1125
        options = "aAc:CdD::fhHI:l:L:m:M:n:p:P:qrsS:UvV-:Y:g:u:x:X"
        arg = <optimized out>
        i = <optimized out>
        ret = <optimized out>
        exit_code = 1
        dont_fork = <optimized out>
        do_help = <optimized out>
        log_set = <optimized out>
        agent_mode = -1
        pid_file = 0x0
        option_compatability = "-Le"
        fd = <optimized out>
        PID = <optimized out>
        __func__ = "main"

The call trace seems similar to a one of the comments in this case
https://github.com/net-snmp/net-snmp/issues/107#issuecomment-886216752

Thanks for the investigation Branko.

Checking the upstream issue you mentioned, it seems that the bug was introduced and fixed after the release of the version we have even in the Ubuntu development release, so I am not sure if this is the reason of your failure. Could you please try the upstream master branch to check if the issue is fixed there?

Hi Lucas,

I'm not able to build latest code from source at the moment, I tried to look for snmpd from backports ..., but no luck. If there are any more details I can get from the coredump I could try to provide them.

Regarding the upstream issue it is really confusing, some people reported using 5.8 (even on Ubuntu), some 5.9. Also what was fixed is not clearly mentioned. So no easy candidates to try and "backport"/patch 5.8.

Thanks,
Branko

Hi Branko,

I prepared a PPA with the new upstream release (amd64 only):

https://launchpad.net/~paride/+archive/ubuntu/net-snmp/

Please let us know if you can reproduce the issue with this one.

Note: some d/patches didn't apply cleanly, so I just removed them from d/p/series. They looked safe to remove (upstreamed, typo fixes and similar), but I didn't do a real check. The packages are experimental builds to test if v5.9.1 has a fix for this, please take them as such.

Hi Paride,

Thank you very much for making the package, unfortunately the server in question is customer production server and have no test server where I can reproduce this issue, I'm not able to test anymore, I had a limited time to solve this, so I did some ugly workaround to prevent daemon from being stopped after it's killed and requiring manual intervention.

By creating a systemd service override:

/etc/systemd/system/snmpd.service.d/override.conf
[Service]
Restart=always

It's not perfect, but still there is a very small chance that service will not be running or crashes while being polled via SNMP.

Maybe other reporter who had these issues on 20.04 can test it. ( @brightdroid )

One of our server's ran into the same issue, snmpd frequently failing with:

snmpd[766]: error on subcontainer 'ifTable container' remove (-1)
kernel: [367589.916607] snmpd[766]: segfault at 91 ip 00007f2af4a20845 sp 00007ffcbd99d3e0 error 4 in libnetsnmpmibs.so.35.0.0[7f2af4986000+d3000]
systemd[1]: snmpd.service: Main process exited, code=dumped, status=11/SEGV
systemd[1]: snmpd.service: Failed with result 'core-dump'.

Running Ubuntu 20.04.6 LTS serving several docker containers. Since there seems to be no permanent solution up until now, I applied the systemd service override. Thanks for the tipp @Branko Grubic.

We're seeing this as well on Ubuntu 22.04:

snmpd[2271493]: error on subcontainer 'ifTable container' remove (-1)
kernel: show_signal_msg: 22 callbacks suppressed
kernel: snmpd[2271493]: segfault at 60 ip 00007f4c9c168423 sp 00007fffa460cdd0 error 4 in libnetsnmpmibs.so.40.1.0[7f4c9c0cc000+d300>Sep 05 06:31:09 co-srv-runnercontainer1 kernel: Code: 55 41 54 49 89 f4 55 53 48 89 fb 48 83 ec 68 4c 8b 2e 48 89 fe 64 48 8b 04 25 28 00 00 00 48 89 44 24 58 48 8b 47 78 systemd[1]: snmpd.service: Main process exited, code=dumped, status=11/SEGV
systemd[1]: snmpd.service: Failed with result 'core-dump'.
systemd[1]: snmpd.service: Consumed 1min 13.654s CPU time.

Combined with a lot of these errors:

snmpd[2006336]: ioctl 35123 returned -1
snmpd[2006336]: ioctl 35123 returned -1
snmpd[2006336]: ioctl 35111 returned -1
snmpd[2006336]: ioctl 35091 returned -1
snmpd[2006336]: ioctl 35105 returned -1
snmpd[2006336]: ioctl 35123 returned -1
snmpd[2006336]: ioctl 35123 returned -1
snmpd[2006336]: ioctl 35111 returned -1
snmpd[2006336]: ioctl 35091 returned -1
snmpd[2006336]: ioctl 35105 returned -1

I don't mind testing any possible fixes, it just might take a few days for them to show up on our system.

Thank you for the update.

Would you be able to provide a coredump, a backtrace and/or a reproducer? I did some research and found upstream's https://github.com/net-snmp/net-snmp/issues/107 which seems to be related, but the patch that fixes their issue doesn't make sense on Jammy's net-snmp, so I'm thinking that there might be more to this bug than meets the eye.

Thank you.

Hey there,

Running ubuntu 22.04, also seeing this issue.

SNMPD version : snmpd/jammy-updates,now 5.9.1+dfsg-1ubuntu2.6 amd64 [installed]

I've got docker containers running on those servers, and I'm using librenms to monitor them.

I've installed systemd-coredump and will follow through when I get one.

BR

Same on Debian bullseye and running docker. But just sometimes and even not on every instance. Strange!

Same problem here in three production servers

 sudo journalctl -u snmpd.service -n 50
Feb 12 16:15:14 accadb001 systemd[1]: Starting Simple Network Management Protocol (SNMP) Daemon....
Feb 12 16:15:14 accadb001 systemd[1]: Started Simple Network Management Protocol (SNMP) Daemon..
Feb 12 16:25:50 accadb001 snmpd[1675639]: ioctl 35123 returned -1
Feb 12 16:25:50 accadb001 snmpd[1675639]: ioctl 35123 returned -1
Feb 12 16:25:50 accadb001 snmpd[1675639]: ioctl 35111 returned -1
Feb 12 16:25:50 accadb001 snmpd[1675639]: ioctl 35091 returned -1
Feb 12 16:25:50 accadb001 snmpd[1675639]: ioctl 35105 returned -1
Feb 12 16:25:50 accadb001 snmpd[1675639]: ioctl 35123 returned -1
Feb 12 16:25:50 accadb001 snmpd[1675639]: ioctl 35123 returned -1
Feb 12 16:25:50 accadb001 snmpd[1675639]: IfIndex of an interface changed. Such interfaces will appear multiple times in IF-MIB.
Feb 12 16:25:50 accadb001 snmpd[1675639]: ioctl 35111 returned -1
Feb 12 16:25:50 accadb001 snmpd[1675639]: ioctl 35091 returned -1
Feb 12 16:25:50 accadb001 snmpd[1675639]: ioctl 35105 returned -1
Feb 12 16:30:53 accadb001 snmpd[1675639]: error on subcontainer 'ifTable container' remove (-1)
Feb 12 16:30:56 accadb001 snmpd[1675639]: error on subcontainer 'ifTable container' remove (-1)
Feb 12 16:30:57 accadb001 systemd[1]: snmpd.service: Main process exited, code=dumped, status=11/SEGV
Feb 12 16:30:57 accadb001 systemd[1]: snmpd.service: Failed with result 'core-dump'.
Feb 12 16:30:57 accadb001 systemd[1]: snmpd.service: Consumed 1.355s CPU time.

Linux accadb001 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy

ii libsnmp-base 5.9.1+dfsg-1ubuntu2.6 all SNMP configuration script, MIBs and documentation
ii libsnmp40:amd64 5.9.1+dfsg-1ubuntu2.6 amd64 SNMP (Simple Network Management Protocol) library
ii snmpd 5.9.1+dfsg-1ubuntu2.6 amd64 SNMP (Simple Network Management Protocol) agents

Hi,

Could you please share the core dump with us? That would help us investigate better this issue.

Sure, attached the crash file from /var/crash.

I'm having the same problem seemingly only on my k8s nodes (running on docker, not containerd), Ubuntu 22.04, and erratic for frequency

snmpd/jammy-updates,now 5.9.1+dfsg-1ubuntu2.6 amd64 [installed]

Can't find a solution anywhere on the internet, so I guess this is me subscribing to this thread

Thanks for the crash report! Looking into the crash report I do think this is related to https://github.com/net-snmp/net-snmp/issues/107. I tried applying the patch to our Jammy version and unfortunately it does not apply completely due to differences in the codebase.

I did apply the lines I could and put them in a PPA[0] if you would like to test this out to see if your issues get resolved, although I'm not confident these changes alone will fix it.

I spent a little trying to reproduce it in a VM and could not find a way to trigger this error, so a reproducer would be very helpful here.

[0] - https://launchpad.net/~mitchdz/+archive/ubuntu/lp720638-snmpd-segv