rpcbind changes after bionic broke rup broadcast feature

Unfortunately I had to time box myself from this issue while doing initial investigation...

I was able to reproduce the issue: after installing openbsd-inetd, tcpd and rstatd + rstat-client I could observe that only xenial and bionic hosts are the ones responding to UDP broadcast requests from "rup" binary:

(c)rafaeldtinoco@xenial:~$ rup
xenial.lxd 20:09 up 1:39, load 0.58 0.63 0.54
bionic.lxd 20:09 up 1:43, load 0.58 0.63 0.54

(c)rafaeldtinoco@bionic:~$ rup
xenial.lxd 20:12 up 1:42, load 0.39 0.50 0.51
bionic.lxd 20:12 up 1:46, load 0.39 0.50 0.51

(c)rafaeldtinoco@eoan:~$ sudo rup
xenial.lxd 20:12 up 1:51, load 0.62 0.52 0.49
bionic.lxd 20:12 up 1:55, load 0.62 0.52 0.49

(c)rafaeldtinoco@focal:~$ rup
xenial.lxd 20:12 up 1:42, load 0.28 0.46 0.50
bionic.lxd 20:12 up 1:47, load 0.28 0.46 0.50

So it is clear that the rpc-rstatd is the one to blame, as all clients can receive info from older ubuntu versions. I was curious because there were practically no changes in all related packages:

- openbsd-inetd
- tcpd (tcp-wrappers)
- rstatd

to justify a change of behavior so I checked syslog in the working nodes:

----
when hostname is given:
xenial rpc.rstatd[2961]: connect from 10.250.97.142 (10.250.97.142)

when broadcast is attempted:
xenial rpc.rstatd[2965]: connect from 127.0.0.1 (127.0.0.1)

and when broadcast is attempted with remote bionic (works):
bionic rpc.rstatd[6264]: connect from 127.0.0.1 (127.0.0.1)
bionic rpc.rstatd[6267]: connect from 127.0.0.1 (127.0.0.1)

----

and there might be something related to UDP broadcast and lo interface...

I backported:

openbsd-inetd 0.20160825-4build1
tcpd 7.6.q-30
rstatd 4.0.1-10

to bionic expecting to brake it and it did not happen =).

As I was running all as containers, on top of the same kernel, it is likely some environmental thing related to how tcp-wrappers are dealing with an UDP socket listening to broadcast.

----

Running inetd by hand (outside systemd scope) did not help also, the broadcast request did not even arrive to the socket as it seems, just the direct one:

(c)rafaeldtinoco@eoan:~$ sudo inetd -idl
pmap_set: 100001 1 17 41139
pmap_set: 100001 2 17 41139
pmap_set: 100001 3 17 41139
pmap_set: 100001 4 17 41139
pmap_set: 100001 5 17 41139
ADD: rstatd rpcprog=100001, rpcvers=5/1, proto=rpc/udp, wait.max=1.256 user:group=nobody:(default) builtin=0 server=/usr/sbin/tcpd
someone wants rstatd
2830 execv /usr/sbin/tcpd
reaping asked for
2830 reaped, status 0
restored rstatd, fd 7
pmap_unset(100001, 1)
pmap_unset(100001, 2)
pmap_unset(100001, 3)
pmap_unset(100001, 4)
pmap_unset(100001, 5)

----

Checking tcpdump:

$ rup eoan.lxd and the eoan container shows:

20:46:39.145376 IP 10.250.97.227.57183 > 10.250.97.200.111: UDP, length 56
20:46:39.145486 IP 10.250.97.200.111 > 10.250.97.227.57183: UDP, length 28
20:46:39.145562 IP 10.250.97.227.51090 > 10.250.97.200.60135: UDP, length 40
20:46:39.150911 IP 10.250.97.200.60135 > 10.250.97.227.51090: UDP, length 132

$ rup with no args and eoan c...

# working env: xenial (without inetd)

*:111 *:* users:(("rpcbind",pid=6679,fd=6))
*:919 *:* users:(("rpcbind",pid=6679,fd=7))
*:998 *:* users:(("rpc.rstatd",pid=6759,fd=3))

(c)rafaeldtinoco@xenial:~$ sudo tcpdump -i eth0 -n
12:23:59.243382 IP 10.250.97.142.38369 > 10.250.97.255.111: UDP, length 100
12:23:59.245356 IP 10.250.97.213.919 > 10.250.97.142.38369: UDP, length 140

# working env: bionic (without inetd)

0.0.0.0:111 0.0.0.0:* users:(("rpcbind",pid=1073,fd=6))
0.0.0.0:825 0.0.0.0:* users:(("rpcbind",pid=1073,fd=7))
0.0.0.0:752 0.0.0.0:* users:(("rpc.rstatd",pid=10753,fd=3))

(c)rafaeldtinoco@bionic:~$ sudo tcpdump -i eth0 -n
11:49:40.673843 IP 10.250.97.227.38276 > 10.250.97.255.111: UDP, length 100
11:49:40.677280 IP 10.250.97.142.825 > 10.250.97.227.38276: UDP, length 140

----

# not-working env: eoan (without inetd)

0.0.0.0:111 0.0.0.0:* users:(("rpcbind",pid=2799,fd=5),("systemd",pid=1,fd=32))
0.0.0.0:901 0.0.0.0:* users:(("rpc.rstatd",pid=2846,fd=3))

(c)rafaeldtinoco@eoan:~$ sudo tcpdump -i eth0 -n
11:54:17.931899 IP 10.250.97.227.48295 > 10.250.97.255.111: UDP, length 100

# not-working env: focal (without inetd)

0.0.0.0:111 0.0.0.0:* users:(("rpcbind",pid=6184,fd=5),("systemd",pid=1,fd=42))
127.0.0.1:862 0.0.0.0:* users:(("rpc.statd",pid=6198,fd=5))
0.0.0.0:39800 0.0.0.0:* users:(("rpc.statd",pid=6198,fd=8))

note: systemd has rpcbind socket, removing that variable had no effect.

----

But I noticed rpcbind is running as _rpc user in the non-working environments.

There was a major change in rpcbind from affected versions.

And… finally:

(c)rafaeldtinoco@groovy:~$ rup
bionic.lxd 12:56 up 18:31, load 0.22 0.16 0.10
groovy.lxd 12:56 up 12:51, load 0.22 0.16 0.10
xenial.lxd 12:56 up 18:26, load 0.22 0.16 0.10

I had to replace groovy rpcbind with rpcbind from bionic.

Now, in some other time, I'll have to check what changes to rpcbind (a lot) made this happen.

Hi. I found that the rpcbind(8) man-page explains a reason for that behavior.
It is included in Bionic, not Focal.

ubu1804:$ man rpcbind
(snip)
-r Turn on remote calls. Cause rpcbind to open up random listening
        ports. Note that rpcinfo need this feature turned on for work
        properly.
(snip)
        Ubuntu note: to retain backwards compatibility with existing behavior,
        in releases Xenial and Bionic, the default for this is enabled,
        so remote calls are turned on by default.
(snip)
        In Focal and later, the remote calls default is disabled, and
        the variable is ignored.

ubu2004:$ man rpcbind
(snip)
FILES
     If "/etc/default/rpcbind" exists, rpcbind will use the specified options
     at launch time rpcbind Otherwise rpcbind will try to load configuration
     from "/etc/rppcbind.conf" rpcbind The default options are set as "-w -f".

So, I edit /etc/default/rpcbind and restart rpcbind.

ubu2004:$ diff -u /etc/default/rpcbind.orig /etc/default/rpcbind
--- /etc/default/rpcbind.orig 2021-09-28 19:10:00.341377024 +0900
+++ /etc/default/rpcbind 2021-09-28 19:10:06.961394965 +0900
@@ -3,7 +3,7 @@
 OPTIONS=""

 # Cause rpcbind to do a "warm start" utilizing a state file (default)
-OPTIONS="-w"
+OPTIONS="-w -r"

ubu2004:$ sudo ss -apn | grep rpcbind | grep ^udp
udp UNCONN 0 0 0.0.0.0:111 0.0.0.0:* users:(("rpcbind",pid=25388,fd=5),("systemd",pid=1,fd=164))
udp UNCONN 0 0 [::]:111 [::]:* users:(("rpcbind",pid=25388,fd=7),("systemd",pid=1,fd=167))
sudo service rpcbind restart
ubu2004:$ sudo service rpcbind restart
ubu2004:$ sudo ss -apn | grep rpcbind | grep ^udp
udp UNCONN 0 0 0.0.0.0:111 0.0.0.0:* users:(("rpcbind",pid=25790,fd=5),("systemd",pid=1,fd=164))
udp UNCONN 0 0 0.0.0.0:56257 0.0.0.0:* users:(("rpcbind",pid=25790,fd=10))
udp UNCONN 0 0 [::]:58194 [::]:* users:(("rpcbind",pid=25790,fd=11))
udp UNCONN 0 0 [::]:111 [::]:* users:(("rpcbind",pid=25790,fd=7),("systemd",pid=1,fd=167))
ubu2004:$ sudo service rpcbind status
(snip)
     CGroup: /system.slice/rpcbind.service
             └─25790 /sbin/rpcbind -f -w -r
ubu2004:$ rusers
ubu2004 testuser
ubu1804 testuser testuser

It seems to be working properly.

Thanks for the further investigation, taraijpn.

Indeed, I was able to (1) reproduce the scenario reported in the description, and (2) confirm that adding the "-r" option to /etc/default/rpcbind's OPTION variable will make the service run as the reporter expected.

I appreciate the time Rafael spent on this bug and his conclusions; it was easy to verify the scenario following his comments.

Having said all that, I consider that this is not a bug, and is properly documented in the manpages. I am therefore setting the status of this bug to Invalid; feel free to change it back to New and to provide more information if you disagree with this decision.

Thank you.

Just for reference/context on why -r isn't enabled by default, and why it's not necessarily a good idea unless you really need it, see bug 304393