openssh-server should ship a systemd generator to generate ssh socket port configuration from sshd_config

Status changed to 'Confirmed' because the bug affects multiple users.

I think a better option is to just not create a socket file for openssh, as it's not required. OpenSSH server is fully capable of managing it's own configuration via a very comprehensive config file, which is universally applicable across various distros and other OSs.

Its just wholly unrequired to allocate an on-demand socket for a daemon that listens all the time, anyways.

A better alternative is to migrate configuration from the openssh configuration file into a systemd unit file, much like how the defaults files provide a baseline set of defaults for things. It can continue to provide defaults, with the unit file overriding the defaults.

A socket for on-demand activation is pretty silly for a daemon that launches, and listens, by design, and returns a status code if unable to launch itself properly. It's just over-complication for the sake of... something? It would also require additional work to make tooling like ansible and puppet do something just for Ubuntu.

I think the biggest issue is the automatic upgrade from "classic" to systemd.socket

It is very hard to consistently migrate every configuration.

I was hit by this issue because I had set /proc/sys/net/ipv6/bindv6only=1, so ipv4 was disabled

If ubuntu wants to use .socket, then please do this only for new openssh-server installations, never on upgrades. You will lock out too many people

@reporter: To be correct the generated configuration would also need to set BindIPv6Only=yes and create explicit 0.0.0.0 and :: (this is what sshd does). Then there is also the AddressFamily option to consider.

All in all way too complex to be done automatically

I'm not sure about this. What's the pattern that users can expect to see across all packages that provide socket activated services in Ubuntu in the future? I think there are three options here: "always", "sometimes" and "never". We could ensure that there's _always_ a generator that creates the corresponding socket unit by doing ad-hoc, probably incomplete parsing of the relevant configuration file; that there _never_ is a generator and users must tweak it by hand; or that there _sometimes_ might be a generator and sometimes not, depending on the service, so manual tweaking may or may not be necessary.

The problem with "always" is the difficulty of parsing the relevant configuration file. So if users start expecting "always" behaviour across all socket activated services across the archive, that's going to be unreliable in a way that we will never be able to fully fix.

"Sometimes" seems horribly inconsistent and unpredictable.

For "never", in https://discourse.ubuntu.com/t/spec-definition-of-an-online-system/27838 I think we agreed that "There will always be cases where if you configure one thing in one place, you’re expected/required to configure another thing in another place to match." I don't think it's unreasonable for the "correct" way to customize an Ubuntu system is to make those other related configuration changes. It's not perfect for UX; neither is any alternative, but at least this way would be consistent and predictable. The pain is in the upgrade path and discoverability of what is required, but we can work to make those things smoother for common use cases. This principle would then apply not only to socket activated services, but also to network-online.target and AppArmor rules that users already have to fix when tweaking system configuration today.

Given the problems with making "always" work in all cases in practice, I think I'd prefer to see us doing what we can as one-off "heuristic" type special handling in the upgrade path in return for a consistent and complete experience for users starting afresh.

> Its just wholly unrequired to allocate an on-demand socket for a daemon that listens all the time, anyways.

The point is to reduce footprint for greater density in production environments. Most instances don't need an ssh daemon running all the time. There's lots of demand from users to make the default installation less "bloated", and I think Ubuntu developers are agreed that one aspect to that is to use socket activation wherever possible. Socket activation for ssh is included in that list.

> If ubuntu wants to use .socket, then please do this only for new openssh-server installations, never on upgrades. You will lock out too many people

I like this idea in principle, but I think it'd just be kicking the can down the road. For example, in a few releases time, we'll inevitably touch something that might break users who upgraded up without converting to socket activation because that will have become a non-obvious configuration. Preventing this would require conditional handling of this special case indefinitely, and maintaining and QA-ing the special case would become impractically difficult over time. Instead, I ...

> The point is to reduce footprint for greater density in production environments. Most instances don't need an ssh daemon running all the time.

If the point is to increase density, then sshd should just be off, and not automatically started, unless it's required for work.

People install openssh-server, so a container/server/whatever has 22 listening on the ports required.

If people need to squeeze 3MB from RAM, a better solution is to just have them never enable it, except when they need it, using a container entry point, or some other hook system that turns it on as required, and then off.

If ssh is selected at install time, to be installed, and listening, then the user expectation is that it is installed, and listening. Not just "listening as needed".

Or, conversely, as I proposed in the original ticket (That somehow got marked as a duplicate of this ticket, that was created later): Migrate all configuration for openssh-server out of /etc/ssh/sshd_config, and into it's unit file.

No, there isn't generally an expectation that you will require two wholly unconnected places to be configured for something that is only configured in one place for every other distro, and every other OS that openssh-server runs on.

One configuration to rule them all, so to speak. And I think at this point, moving config for openssh-server into a unit file, is probably the best course.

On Thu, Oct 06, 2022 at 01:04:05PM -0000, Corey Reichle wrote:
> If the point is to increase density, then sshd should just be off, and
> not automatically started, unless it's required for work.

Socket activation provides a smoother (runtime) UX for users, and is
well established as a mechanism to reduce runtime footprint without any
impact to users who want to use the service. Why do you think it's
preferable to have the daemon not started and without socket activation?

> If ssh is selected at install time, to be installed, and listening, then
> the user expectation is that it is installed, and listening. Not just
> "listening as needed".

Why? What user story is broken by socket activation here?

> Or, conversely, as I proposed in the original ticket (That somehow got
> marked as a duplicate of this ticket, that was created later): Migrate
> all configuration for openssh-server out of /etc/ssh/sshd_config, and
> into it's unit file.

I'm pretty sure this would result in far more pushback from the
community than merely enabling socket activation. We'd end up with an
order of magnitude more upgrade path issues in doing this, and we'd be
diverging from the entire rest of the community.

> No, there isn't generally an expectation that you will require two
> wholly unconnected places to be configured for something that is only
> configured in one place for every other distro, and every other OS that
> openssh-server runs on.

It's increasingly common to use socket activation on systemd-based
distros. Ubuntu may be pushing ahead on the sshd side, but socket
activation in general is already in place in various other packages.

I accept that the "two different places" configuration issue arises as a
consequence of socket activation, and this is poor UX. But the general
concept already exists in other areas (eg. After=network-online.service,
and AppArmor), and doing otherwise in the general case would require a
reversal, or even a "ban", on the use of socket activation across all
packages in Ubuntu. I don't think that makes sense, but even if it did,
it'd have to be a bigger discussion than just in this bug. As long as
socket activation is a generally acceptable pattern in Ubuntu, I see no
reason why sshd would be expected to be special and not use it.

> Socket activation provides a smoother (runtime) UX for users

SSHD configuration is not a user issue, but a systems administration issue. A smoother UX for system administrators is a) Fully documented solutions, or b) One source of truth for all things regarding a service, hence the push for infrastructure as code - The repo is the one source of truth for the service.

> Why do you think it's preferable to have the daemon not started and without socket activation?

Because, when a systems administrator, or devops person decides to enable SSH listening... they expect ssh to be listening. And they both expect the documented configuration of openssh to be standard across everywhere openssh-server is installed.

> Why? What user story is broken by socket activation here?

The user story that prompted this ticket:

I configured ssh to listen on one family, and one address, with another service listening on 22 on another address, for both families.

Regardless of my nginx configuration and my openssh-server configuration, unbeknownst to me... a socket file was dropped for openssh-server, making nginx failing to start, because it could not bind to the required ports.

Nowhere was it documented that sshd now needs two configuration files, just to tell it to listen on a port, and address it was already told to do. openssh-server when started didn't mention that it's configure on a port, that requires a socket. And my only way to intuit it was to find init listening on 22 for some god awful reason.

> I'm pretty sure this would result in far more pushback from the community than merely enabling socket activation.

Are we sure about this? Pretty certain the push to systemd for the linux world, would make it pretty easy to figure out the configuration is now in a unit file, where it should be anyways. Just like how people now know that /etc/network/interfaces it not the place to configure the network now, but rather /etc/netplan/*.yaml. Its documented, and the way.

> We'd end up with an order of magnitude more upgrade path issues in doing this, and we'd be diverging from the entire rest of the community.

This design choice already diverges from the rest of the community. This is the only distro where a very common service has undocumented methods of configuring ports. It's almost like there's a push to re-create inetd here, without making the required changes to use such a supervisor service.

> or even a "ban", on the use of socket activation across all packages in Ubuntu

I'm not proposing a ban on socket activation, even for this package. Just one place to configure the service, not two, or three.

> As long as socket activation is a generally acceptable pattern in Ubuntu, I see no reason why sshd would be expected to be special and not use it.

If Ubuntu is going the route of multiple places to configure everything, then perhaps the problem is me, and I need to search for another solution over Ubuntu. Because two and three sources of truth is not optimal, especially when undocumented, at all. Regardless of an "accepted pattern".

On Thu, Oct 06, 2022 at 02:34:40PM -0000, Corey Reichle wrote:
> > Socket activation provides a smoother (runtime) UX for users
>
> SSHD configuration is not a user issue, but a systems administration
> issue.

Maybe, but more important is the UX for users of ssh clients who aren't
trying to deviate from default configuration, and for users who want
density by default and don't need sshd running on every instance. Socket
activation provides a smoother UX for these user stories.

It sounds like your objections relate to the use of socket activation
and how it is implemented in configuration files in general, rather than
ssh specifically. The same goes for your ideas about moving daemon
configuration into systemd unit files. I think your views on both of
these issues contradict the general direction that our entire ecosystem
is taking. Essentially you seem to be asking for a complete reversal in
direction of how socket activation is used and configured in Debian and
Ubuntu.

You're entitled to your opinions and I wouldn't want to rule anything
out, but this is the wrong venue for these discussions as they are far
wider reaching than just ssh. I suggest you use
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss instead.

On Thu, Oct 06, 2022 at 02:34:40PM -0000, Corey Reichle wrote:
> Nowhere was it documented that sshd now needs two configuration files,
> just to tell it to listen on a port, and address it was already told to
> do.

False. It is documented in the manpage, in the default sshd_config file
shipped in the package, and in the kinetic release notes.

> False. It is documented in the manpage,

I looked in the manpage for my install, and no mention of socket activation.

I looked in the README.Debian, and it indicated you can switch to socket activation, over standard daemonization, if you choose, by masking sshd, and enabling sshd.socket, and starting sshd.socket instead.

Maybe it's in the new release. I am basing my info on where I encountered this issue.

On Thu, Oct 06, 2022 at 07:32:54AM -0000, msaxl wrote:
> I think the biggest issue is the automatic upgrade from "classic" to
> systemd.socket

> It is very hard to consistently migrate every configuration.

> I was hit by this issue because I had set
> /proc/sys/net/ipv6/bindv6only=1, so ipv4 was disabled

Please file a separate bug explaining exactly what problem you encountered
on upgrade. Setting /proc/sys/net/ipv6/bindv6only=1 (which is not something
I consider a sensible deviation from the Ubuntu defaults) should not have
caused you to experience upgrade issues, and is independent from the
question of using a generator.

This is fixed now right?

I believe this is fixed in:

openssh (1:9.6p1-3ubuntu3) noble; urgency=medium

  * Add sshd-socket-generator to generate ssh.socket drop-in configuration
  [...]

From README.Debian:

The provided ssh.socket unit file sets ListenStream=22. A systemd generator,
sshd-socket-generator, parses the sshd configuration and, if non-default
ListenAddress(s) or Port(s) are configured, generates corresponding
ListenStream= lines in a drop-in configration file for ssh.socket.

So marking this as Fix Released.