systemd-logind network access is blocked, and breaks remote authentication configurations

The Debian bug is marked as fixed, however unfortunately it seems to have 'fixed' the problem simply by adding Recommends: nscd to the ypbind-mt package, which only actually fixes things if the systemd admin does install/use nscd. If the admin chooses not to use nscd, this bug still exists.

Status changed to 'Confirmed' because the bug affects multiple users.

Other than the obvious approach of enabling systemd-userdb for Ubuntu, which is a much larger discussion/decision, I think there are really only 2 ways to address this:

1) Include drop-in conf files for systemd-logind and systemd-udevd to remove the networking sandbox
2) add configuration documentation to nis and openldap instructing the system admin to create drop-in conf files for systemd-logind and systemd-udevd as part of system configuration

Option #1 has the advantage of 'just working' without any local admin changing anything, but has the disadvantage of completely removing network sandboxing for logind/udevd.

Option #2 has the advantage of keeping the sandboxing and allowing the admin to customize it more specifically, such as allowing networking only to specific nis/ldap servers instead of allowing all networking, but has the disadvantage of requiring the system admin to read the docs and actually perform the additional configuration.

I'm skeptical of option #1 as the network sandboxing is a security feature, but also I'm pretty sure if we go with option #2 there will be plenty more bugs opened due to admins missing that part of the local system configuration.

Any opinions or other ideas on approaches?

1) Include drop-in conf files for systemd-logind and systemd-udevd to remove the networking sandbox

Those drop-in configs should be shipped in the nis package. I don't see a reason to ship a drop-in for systemd-udevd, fwiw.

> Other than the obvious approach of enabling systemd-userdb for Ubuntu,

I don't see how that would help, given that sytemd-userdb.service has

RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6

You basically have the same issue as with systemd-logind.service. Or am I missing something here?

Status changed to 'Confirmed' because the bug affects multiple users.

Xubuntu-20.04, 20.10 and 21.04 are not usable with ldap authentication - very bad!
Not usable for us.

> > Other than the obvious approach of enabling systemd-userdb for Ubuntu,
>
> I don't see how that would help, given that sytemd-userdb.service has
>
> RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6
>
> You basically have the same issue as with systemd-logind.service. Or am I missing something here?

I may be misunderstanding how upstream intends it all to work, but I believe that since the userdb service does include AF_INET/AF_INET6 in RestrictAddressFamilies, those are *allowed* families for the userdb service. The naming of the parameter doesn't seem great to me, at first read it's hard to understand if the assigned families are *allowed* or *restricted*...but I'm pretty sure the assigned families are *allowed* and all other (unlisted) families are *restricted* (blocked), meaning userdb is allowed to make inet/inet6 connections, unlike logind, which has only:

RestrictAddressFamilies=AF_UNIX AF_NETLINK

Just to summarize the specific flow of this bug:

1. an application is started for a user session, e.g. sshd handles a user connecting.
2. the application uses pam for authentication, which by default includes pam_systemd as an (optional) module.
3. pam invokes pam_systemd as part of session creation.
4. pam_systemd calls into systemd-logind, over dbus with CreateSession.
5. the running systemd-logind process receives the dbus request and begins setting up the session.
6. systemd-logind attempts to get the full user info via NSS, e.g. getpwuid_r() or similar call
7. glibc attempts the user lookup, doesn't find the UID or username in /etc/passwd using the normal "files" based lookup
8. since nsswitch.conf is configured with network-based authentication (an essential part of this bug), glibc attempts (either through its "compat" mechanism, or separate nss module like libnss_ldap) to open a network connection to the remote authentication server to get the user info
9. since the glibc code, as well as any libnss module loaded by glibc, is running under the systemd-logind process, the network connection fails due to systemd-logind RestrictAddressFamilies= setting
10. nss returns a result of "no user found" to systemd-logind
11. systemd-logind determines it was unable to find the user data for the requested session, so it returns failure over dbus to the pam_systemd module
12. the pam_systemd module returns failure, but since it's marked "optional" (by default) in the pam common-session config file, the application isn't stopped from finishing successful auth
13. the application finishes pam authentication and opens a shell, but without a successful systemd-logind session

note that if there is a local caching component to whatever network authentication is configured - e.g. nscd - glibc might attempt to check that locally first, and if the user is located and no remote connection is needed, the lookup will succeed.

also note that this bug only applies for nss modules that actually perform network connections directly from the nss module; if the nss module calls a local unix socket into a separate locally running process, that won't be blocked, since systemd-logind is allowed to do AF_UNIX.

As has been pointed out and discussed, it really doesn't seem like there are any options to address this other than 1) open up systemd-logind to AF_INET and/or AF_INET6, or 2) provide systemd-logind with a way to perform nss lookups outside of direct nss calls, e.g. userdb.

I think Dan's summary above is very good. For clarification I would add a couple of points.

The issue is not just remote logins. xdm behaves in the same way, and the absence of a systemd-logind session may mean that sound is then unavailable to the user logged in at the console. (Mentioned to help people searching for local sound issues.)

Comments 12 and 16 of bug #1915502 also mention ProtectHostname=no which I don't understand.

My understanding of nscd is that, even on cache misses, it will perform the lookup itself, and, being a separate process outside the systemd-logind sandbox, it will succeed. I am not convinced that mandating the use of nscd would be a good idea though, especially as some distributions are moving away from it, e.g. https://fedoraproject.org/wiki/Changes/RemoveNSCD I suspect a lot of NIS/LDAP users do use some version of nscd, which is why there are not more people caught by this issue.

@Dan: have you actually confirmed, that building and running userdbd solves those issues with NIS and LDAP?

I have to add, that I don't have such a NIS or LDAP setup to test this myself.

> @Dan: have you actually confirmed, that building and running userdbd solves those issues with NIS and LDAP?

sorry for the delay in getting back to this.

So, you're correct, userdb doesn't actually help this, it only moves the problem.

While systemd-userdbd.service does (currently, at least) allow AF_INET and AF_INET6, I missed that it also includes IPAddressDeny=any which results in the problem existing there as well.

So if we include systemd-userdbd in Debian/Ubuntu, then the network-restricted systemd-logind problem is simply moved to systemd-userdbd. I will note that I did verify, if systemd-logind is running with its default restrictions and systemd-userdbd is also running without its IPAddressDeny=any restriction, the issue is fixed; systemd-logind does correctly get the user record from NIS through systemd-userdbd.

I also seem to have misunderstood Lennart's position on this; after re-reading his statements[1] it seems that his position is that *all* systemd services should be locked down (or at least, both systemd-logind and systemd-userdbd) and the responsibility for poking holes in one (or both, or more) of those services' restrictions is on NIS and any other NSS library that requires network access.

So enabling systemd-userdbd won't help with this bug, unless we want to diverge from upstream by removing its IPAddressDeny=any restriction. But we could just as easily remove that from systemd-logind (along with adding AF_INET and AF_INET6 to it).

[1]:
https://github.com/systemd/systemd/issues/7074#issuecomment-338157851
https://github.com/systemd/systemd/issues/15705#issuecomment-624125354

One good point in favor of including systemd-userdbd in Debian/Ubuntu would be that we only need to adjust the restrictions for that service; all other systemd-provided services would use (or at least, *should* use...) systemd-userdbd to get user records. I'm not sure if that is actually worth including and using systemd-userdbd to the distro.

Putting that aside for now, I think the only options to fix this all involve removing the systemd-logind restriction(s). The only question seems to be what/who removes the restrictions.

1) We could simply remove the restrictions for systemd-logind in the Debian/Ubuntu package, for everyone. This would (theoretically) reduce the security of systemd-logind by allowing it to access the network, but also it would allow network-based NSS lookups to work in all situations (meaning, for all possible NSS modules, not only NIS and/or select other packages/modules).

2) We could add drop-in files to remove the restrictions for systemd-logind, for the 'nis' package (which I think has been split out in later releases and in Debian, so we'd need to pick the correct client package). We would also need to include drop-in files for any other package that provides an NSS module that needs to perform network lookups, such as the 'libnss-ldap' package (I haven't verified this needs network access but I believe it does, e.g. comment 7 above).

3) We could simply add documentation to the README and/or man pages and/or other docs and leave it up to the system admin to add the systemd-logind drop-in file when they set up NIS or LDAP or whatever other network-based auth provider.

Personally, I think #3 is barely better than the current situation; there's just far too many setup guides and other documentation out there for us to expect all Debian/Ubuntu users to notice the additional step we might add to our own docs, even if it's in the package README and man pages. However, if we think that NSS-based network auth services are a thing of the past (as seems to be the upstream position on this), then maybe just adding docs is the right way to handle this.

I'm not too enthusiastic about option #2 either. While it would help with users of specific packages, without those users needing to know any specific details about the workaround, it won't cover all use cases - since we don't necessarily know all the different packages that might need to provide a drop-in - and also it could *add* confusion in some cases. For example, if someone is using a custom network-based libnss module, it would work on any system where they had the 'nis' package installed, but fail on any system where it wasn't installed - regardless of whether they are actually using nis for anything at all. That's the kind of thing that is incredibly painful to debug. This option also would require the 'nis' and 'libnss-ldap' packages to restart systemd-logind when they are installed, which (in the past at least) has sometimes been problematic.

I know that option #1 is kind of the 'worst', since it essentially just gives up and removes the restriction from systemd-logind for everyone. However, I personally think it's the best option. If/when we do start providin...

I initially preferred your option two, a drop-in file in whichever nis
and ldap binary packages, on principle of trying to keep the mitigations
in place if we can.

But your case for a difficult debugging session is persuasive. Reading
the various bug reports around this, option three seems pretty bad --
none of those symptoms would make me think of changing a systemd hardening
configuration on a service I might not know I am running. Nothing really
looked obviously related to network-based id services. Trying to provide
documentation around that won't be very discoverable.

Ubuntu is supposed to be easy.

So, option one: removing the restrictions for systemd-logind in our
package.

It would be nice if our implementation of option one would make it very
easy to re-add the hardening setting; which we could then document in a
hardening guide.

Thanks

Ok let's go with option #1 then, just open up systemd-logind to network access directly by editing the service file.

@mbiebl, do you want to patch this in Debian too, should I open a merge request in salsa? Obviously if Debian is patched first, that's ideal. Assuming you're ok with directly patching systemd-logind to open network access, instead of putting a drop-in file in the nis package.

btw, I no longer work for Canonical, and this bug doesn't personally affect me, so it's unlikely I will follow up on this; if anyone does care about this bug, please feel free to take this over