package openssh-server 1:9.0p1-1ubuntu7 failed to install/upgrade: postinstall script returned 1

sshdconfig.txt actually does NOT contain a line with Port 22 (it is commented out)

the beginning of the file is:

Include /etc/ssh/sshd_config.d/*.conf

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
ListenAddress [fd12:2017:8387:80:3897:7aff:fe15:4de6]:22
ListenAddress [::]:15902
ListenAddress 0.0.0.0:15902

> sshdconfig.txt actually does NOT contain a line with Port 22 (it is commented out)

Why do you mention this? I don't see anything in the log file that mentions Port settings.

Your dpkg log shows:

> Failed to restart ssh.socket: Unit ssh.socket has a bad unit file setting.
> See system logs and 'systemctl status ssh.socket' for details.
> Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
> dpkg: Fehler beim Bearbeiten des Paketes openssh-server (--configure):
> »installiertes post-installation-Skript des Paketes openssh-server«-Unterprozess gab den > Fehlerwert 1 zurück

Can you run this 'systemctl status ssh.socket' command and attach the output?

For developer reference, the logs show this is an upgrade from the jammy version of openssh-server package to 1:9.0p1-1ubuntu7 in kinetic.

Based on what you quote of the contents of your sshd_config, my expectation would be that we NOT migrate to socket activation on upgrade.

> Why do you mention this? I don't see anything in the log file that mentions Port settings.

Because the automatically attached file has that setting and thus does not reflect my real configuration

anyway, a issue seems to be that hostnames_to_addresses does not handle
ListenAddress 1.2.3.4:1234, only ListenAddress 1.2.3.4 (without port)

addresses=$(hostnames_to_addresses "$addresses")
empties/breaks the list in my case, but somehow this addresses.conf is created:

[Socket]
ListenStream=

Hellow, same issue here. SSHD failed during the do-release-upgrade process on one of my machines, and now it refuses to change the port to anything else than 22 on both of my machines. If I start the sshd daemon using systemd service -> systemctl start ssh.service

ssh does change the port if I load the config using the -f command after manually launching the binary.

/usr/sbin/sshd -f /path/to/my/config.conf

Some additional info:

drop in settings from sshd.config.d seem to be applied normally, the issue seem to be only for IP binding and custom ports.

If I change Accept=no by Accept=yes in ssh.socket and reloads the socket unit, I can start sshd on a different port and I can also bind the IP to something else than ::

There's an issue still, an instance of sshd is still listening to :::22 that is not started by SSHD but by init.

root@ubuntulocal:~# netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 568/vsftpd
tcp 0 0 0.0.0.0:622 0.0.0.0:* LISTEN 571/sshd: /usr/sbin
tcp 0 272 192.168.1.225:622 192.168.1.220:2473 ESTABLISHED 1027/sshd: root@pts
tcp6 0 0 :::22 :::* LISTEN 1/init

If I reboot after changing this no to yes in ssh.socket does not survive a reboot and fails to load sshd with a "Failed to queue service startup job" error.
Oct 21 15:41:56 ubuntulocal systemd[1]: ssh.socket: Failed to queue service startup job (Maybe the service file is missing or not a template unit?): Invalid argument
Oct 21 15:41:56 ubuntulocal systemd[1]: ssh.socket: Failed with result 'resources'.

I had to mask/stop the sshd.socket unit and create a custom sshd service in /etc/systemd/system to be able start sshd on a custom port and IP.

On Fri, Oct 21, 2022 at 02:06:33PM -0000, Christophe M. wrote:
> Hellow, same issue here. SSHD failed during the do-release-upgrade
> process on one of my machines, and now it refuses to change the port to
> anything else than 22 on both of my machines. If I start the sshd daemon
> using systemd service -> systemctl start ssh.service

Please file a separate bug report for your upgrade issue, including upgrade
logs. It is not clear that what you are describing is the "same issue".

It is exactly the same issue msaxl reported, the Opensshd returns an error 1 at the ends of the upgrade process, and after the upgrade it is impossible to bind/listen opensshd on anything else than :::22.

The first issue which is that opensshd reported an error 1 at the end of the upgrade process, happened on one machine.

I'll open a ticket for the bind/port issue as msaxl isn't really clear with what he said regarding the behavior of Opensshd post-upgrade, and I understood only because I had the exact same issue.

The second that is about the sshd.socket, or whatever it is, unit that doesn't allow opensshd to listen or bind to anything other than :::22. In short it is impossible to bind or listen to a custom port with Opensshd after upgrading 22.04 to 22.10 on both of my machines. The machine that returned an error 1 and the other that didn't. Same as msaxl

That second issue that msaxl reported happens to me on both of my machine, including the machine that did not report an opensshd error 1 at the end of the upgrade process.

I joined the log from the machine that had the opensshd error 1

2022-10-21 14:59:29,800 ERROR got an error from dpkg for pkg: 'openssh-server': 'installed openssh-server package post-installation script subprocess returned error exit status 1'
2022-10-21 14:59:29,801 DEBUG running apport_pkgfailure() openssh-server: installed openssh-server package post-installation script subprocess returned error exit status 1
2022-10-21 15:00:17,041 ERROR Exception during pm.DoInstall()

slightly off-topic for those who find this before the 22.10 documentation:
https://discourse.ubuntu.com/t/sshd-now-uses-socket-based-activation-ubuntu-22-10-and-later/30189

This bug is about the postinst script not being able to convert (or keep) every configuration around.

In my case this is because the script is unable to parse ListenAddress with port. The script should understand this formats (quote from man sshd_config):

ListenAddress hostname|address
ListenAddress hostname:port
ListenAddress IPv4_address:port
ListenAddress [hostname|address]:port

only the first is supported (and [hostname|address] without port)
example of working statements
ListenAddress localhost
ListenAddress 127.0.0.1
ListenAddress [::1]
ListenAddress ::1

example of valid but not working statements:
ListenAddress localhost:2222
ListenAddress 0.0.0.0:2222
ListenAddress [::]:2222

When converting [::] I think BindIPv6Only=yes should be used, but that's another topic

Port
Specifies the port number that sshd(8) listens on. The default is 22. Multiple options of this type are permitted. See also ListenAddress. Note: On Ubuntu, the openssh-server package is configured to use systemd socket-based activation by default. Therefore if you are using systemd with the default configuration, Port options will not be honored. Address configuration must be handled in /etc/systemd/system/ssh.socket.d instead.

# Port and ListenAddress options are not used when sshd is socket-activated,
# which is now the default in Ubuntu. See sshd_config(5) and
# /usr/share/doc/openssh-server/README.Debian.gz for details.
------------------------------

For me, the post upgrade error didn't happen on the machine that had the default openssh-server port 22, but on the machine that had a custom port and bind address. I had to purge and reinstall openssh-server to fix the broken, partially installed, package post-upgrade.

I only noticed the new message in the sshd_config configuration file after purging/reinstalling openssh-server second time on one of my machine. I was multitasking during the upgrade process, if there was a message about the new port configuration being a systemd socket, I missed it.

@crhis34 actually socket activated ssh is not that difficult to setup. The issue for me is only that the upgrade of the configuration did not work quite right (and I think it is really challenging to do that right)

but simply put you take the file
/etc/systemd/system/ssh.socket.d/addresses.conf

and write something like
[Socket]
ListenStream=
ListenStream=4444

(note the empty ListenStream=, that one is required to not listen on port 22)
followed by a systemctl daemon-reload

then if there is no /etc/systemd/system/ssh.service.d created by the upgrade script then what I did was
systemctl enable ssh.socket && systemctl disable --now ssh.service && systemctl start ssh.socket

an alternative might be doing that what the update script does:
override_dir=/etc/systemd/system/ssh.service.d
mkdir -p "$override_dir"
echo '[Unit]' > "$override_dir"/00-socket.conf
echo 'After=ssh.socket' >> "$override_dir"/00-socket.conf
echo 'Requires=ssh.socket' >> "$override_dir"/00-socket.conf

# deb-systemd-helper is inadequate for the task of
# changing policy for the units on upgrade
if [ -d /run/systemd/system ]; then
       systemctl daemon-reload
       systemctl disable ssh.service
       systemctl unmask ssh.service
       systemctl stop ssh.service
       systemctl enable ssh.socket
fi

I deleted my account, so I had to create a new one. I used Linux headless, I'll move away to a Systemd Linux distro that doesn't pull shenanigans or tries to reinvent the wheel every update.

Whomever pushed this script didn't test it with anything else than the default all commented ListenAddress and Port in the config. On my install it failed to convert a custom Listenaddress and Port formatted exactly as in the default sshd_config file:

Port 622
ListenAddress 0.0.0.0

The machine that didn't return an openssh-server error post-upgrade had and almost untouched sshd_config with ListenAddress and Port commented to use opensshd-server default ListenAddress/Port.

I know how to set ports with sockets. I didn't see the new # comment in the sshd_config at first I just didn't get why the third party systemd socket was taking precedence over the config from the original application package. The machine that didn't fail installing openssh-server post-upgrade had a /etc/systemd/system/ssh.socket.d/override.conf set.

Sorry if I hijacked your bug report a bit.

The attachment "openssh_9.0p1-1ubuntu8.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

+ if dpkg --compare-versions "$2" lt-nl 1:9.0p1-1ubuntu8~ && [ -n "$NO_SOCKET_MIGRATION" ]; then

I'm going to ask that we be ultra-conservative here. It is very difficult in general to un-do in a maintainer script something that we think we did previously, because the admin may have done who-knows-what in between and we may be undoing things that weren't actually ours. So whenever it's possible to detect that it wasn't us that did a thing, we should avoid trying to undo it.

In the case of this particular failure, the state of the system of a user hit by this bug will be:
 - /etc/systemd/system/ssh.service.d/00-socket.conf and /etc/systemd/system/ssh.socket.d/addresses.conf both exist
- $2 argument to postinst will be LESS than 1:9.0p1-1ubuntu8~ because the release version of openssh-server will have failed to configure

So I suggest the following instead:

        if dpkg --compare-versions "$2" lt-nl 1:9.0p1-1ubuntu7~ \
           && [ -e /etc/systemd/system/ssh.socket.d/addresses.conf ] \
           && [ -e /etc/systemd/system/ssh.service.d/00-socket.conf ] \
           && [ -n "$NO_SOCKET_MIGRATION" ]; then

This ensures that if, for any other reason the user has enabled the ssh.socket unit but our script says NO_SOCKET_MIGRATION, we don't mangle the systemd units to disable socket activation that might not have been enabled by us in the first place.

(As a bonus, it will simplify and shorten the de-migration code overall.)

An upload of openssh to kinetic-proposed has been rejected from the upload queue for the following reason: "patch to be revised".

This revised patch addresses Steve's review comments.

Hello msaxl, or anyone else affected,

Accepted openssh into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:9.0p1-1ubuntu7.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted openssh (1:9.0p1-1ubuntu7.1) for kinetic have finished running.
The following regressions have been reported in tests triggered by the package:

dropbear/2022.82-4 (armhf)
gvfs/1.50.2-2 (amd64, arm64)
sbuild/0.83.1ubuntu1 (s390x, amd64, arm64, ppc64el)
piuparts/1.1.5 (s390x, amd64, arm64, ppc64el)
xen-tools/4.9.1-1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/kinetic/update_excuses.html#openssh

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

I have verified each test case using openssh-server 1:9.0p1-1ubuntu7.1 from kinetic-proposed:

Test #1:

root@jammy:~# grep "^ListenAddress" /etc/ssh/sshd_config
ListenAddress 0.0.0.0:1234
root@jammy:~# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: active (running) since Thu 2022-11-03 10:22:04 UTC; 30s ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 868 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 869 (sshd)
      Tasks: 1 (limit: 18901)
     Memory: 1.7M
        CPU: 19ms
     CGroup: /system.slice/ssh.service
             └─869 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

Nov 03 10:22:04 jammy systemd[1]: Starting OpenBSD Secure Shell server...
Nov 03 10:22:04 jammy sshd[869]: Server listening on 0.0.0.0 port 1234.
Nov 03 10:22:04 jammy systemd[1]: Started OpenBSD Secure Shell server.
root@jammy:~# vi /etc/apt/sources.list
root@jammy:~# cat /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu kinetic main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu kinetic-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu kinetic-proposed main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu kinetic-security main restricted universe multiverse
root@jammy:~# apt update && apt dist-upgrade -y
[...]
root@jammy:~# cat /etc/systemd/system/ssh.socket.d/addresses.conf
[Socket]
ListenStream=
ListenStream=0.0.0.0:1234
root@jammy:~# systemctl status ssh.socket
● ssh.socket - OpenBSD Secure Shell server socket
     Loaded: loaded (/lib/systemd/system/ssh.socket; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/ssh.socket.d
             └─addresses.conf
     Active: active (listening) since Thu 2022-11-03 10:31:12 UTC; 23s ago
      Until: Thu 2022-11-03 10:31:12 UTC; 23s ago
   Triggers: ● ssh.service
     Listen: 0.0.0.0:1234 (Stream)
      Tasks: 0 (limit: 18901)
     Memory: 8.0K
        CPU: 332us
     CGroup: /system.slice/ssh.socket

Nov 03 10:31:12 jammy systemd[1]: Listening on OpenBSD Secure Shell server socket.
---
Test #2:

root@jammy:~# grep "^ListenAddress" /etc/ssh/sshd_config
ListenAddress 0.0.0.0:1234
ListenAddress [::]:4321
root@jammy:~# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: active (running) since Thu 2022-11-03 10:33:34 UTC; 4s ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 868 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 869 (sshd)
      Tasks: 1 (limit: 18901)
     Memory: 1.7M
        CPU: 42ms
     CGroup: /system.slice/ssh.service
             └─869 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

Nov 03 10:33:34 jammy systemd[1]: Starting OpenBSD Secure Shell server...
Nov 03 10:33:34 jammy sshd[869]: Server listening on...

I've checked that it did not do the rollback if I manually enabled ssh.socket with the "configuration" ssh.socket.d/00-sockets.conf (and addresses.conf missing)

LGTM

The autopkgtest failures in kinetic have all been resolved with retries and/or hinting.

This bug was fixed in the package openssh - 1:9.0p1-1ubuntu7.1

---------------
openssh (1:9.0p1-1ubuntu7.1) kinetic; urgency=medium

  * debian/openssh-server.postinst: Fix handling of ListenAddress when a port
    is specified (LP: #1993478):
    - Strip port before converting hostnames to numerical addresses.
    - Only append ports when the ListenAddress does not already specify a
      port.
    - Revert socket migration on upgrade if a previous version did the
      migration when it should not have.
  * debian/openssh-server.postinst: Ignore empty directory failure from rmdir
    when skipping socket migration (LP: #1995294).

 -- Nick Rosbrook <email address hidden> Tue, 25 Oct 2022 11:57:43 -0400

The verification of the Stable Release Update for openssh has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package openssh - 1:9.0p1-1ubuntu8

---------------
openssh (1:9.0p1-1ubuntu8) lunar; urgency=medium

  * debian/openssh-server.postinst: Fix handling of ListenAddress when a port
    is specified (LP: #1993478):
    - Strip port before converting hostnames to numerical addresses.
    - Only append ports when the ListenAddress does not already specify a
      port.
    - Revert socket migration on upgrade if a previous version did the
      migration when it should not have.
  * debian/openssh-server.postinst: Ignore empty directory failure from rmdir
    when skipping socket migration (LP: #1995294).

 -- Nick Rosbrook <email address hidden> Tue, 25 Oct 2022 11:57:43 -0400