socket-activated sshd breaks on concurrent connections
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
openssh (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
Focal |
Fix Released
|
Undecided
|
Athos Ribeiro | ||
Hirsute |
Fix Released
|
Undecided
|
Athos Ribeiro |
Bug Description
[Impact]
Users of the systemd socket activated ssh service may experience a race condition that may lead an ssh instance to fail.
The race condition happens when, for a running socket activated ssh service,
an instance A is started, creating the RuntimeDirectory for the service; then
an instance B is started, relying on the RuntimeDirectory created for instance A; then
instance A halts, causing the RuntimeDirectory to be deleted.
If, at this point, instance B has not chrooted into RuntimeDirectory yet, then instance B will fail.
The proposed patch fixes the issue by preserving the RuntimeDirectory after an instance A of the socket activated ssh service halts.
[Test Plan]
1) Stop any running instances of ssh.
`systemctl stop ssh`
2) Start the socket activated ssh service.
`systemctl start ssh.socket`
3) Verify that no errors related to ssh were logged in /var/log/auth.log
`cat /var/log/auth.log | grep 'sshd.*
4) perform several ssh connections to the running server in a short time span. ssh-keyscan may help here.
`ssh-keyscan localhost`
5) Verify that errors related to ssh were logged in /var/log/auth.log
`cat /var/log/auth.log | grep 'sshd.*
6) Apply the proposed fix (make sure the socket activated service is restarted)
7) repead step (4), then verify that no new entries were appended to the step (5) output
[Where problems could occur]
If the changes to the socket activated unit file are wrong, the socket activated service may fail to start after the package upgrade. In this case, we would need to instruct users to perform local changes to the unit file with possible additional fixes while a new version of the patch lands.
[racb] There might be cases where users are inadvertently depending on the cleanup that will now be disabled - for example by a bug or misconfiguration that would result in /run filling up otherwise. By disabling systemd cleanup and relying solely on openssh for cleanup, such a bug or misconfiguration may be exposed and cause problems on such systems.
[Other Info]
This fix has been forwarded to Debian and accepted in https:/
[Original message]
This is mostly the same issue as https:/
With the default configuration of openssh-server and systemd, sshd will complain and crash when multiple connections are made and terminated in a quick succession, e.g. with `ssh-keyscan`. It results in the following errors in /var/log/auth.log:
```
Nov 22 20:53:34 {host} sshd[14567]: Unable to negotiate with {client} port 41460: no matching host key type found. Their offer: <email address hidden> [preauth]
Nov 22 20:53:34 {host} sshd[14570]: fatal: chroot(
Nov 22 20:53:34 {host} sshd[14569]: fatal: chroot(
Nov 22 20:53:34 {host} sshd[14568]: fatal: chroot(
Nov 22 20:53:34 {host} sshd[14566]: fatal: chroot(
Nov 22 20:53:47 {host} sshd[14584]: Connection closed by {client} port 59312 [preauth]
Nov 22 20:53:47 {host} sshd[14586]: fatal: chroot(
Nov 22 20:53:48 {host} sshd[14585]: fatal: chroot(
```
as well as e.g. missing responses in ssh-keyscan:
```
$ ssh-keyscan -vvv {host}
debug2: fd 3 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 2
debug2: fd 4 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 4
debug2: fd 5 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 8
debug2: fd 6 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 32
debug2: fd 7 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 64
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-
debug2: host key algorithms: <email address hidden>
debug2: ciphers ctos: <email address hidden>
debug2: ciphers stoc: <email address hidden>
debug2: MACs ctos: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>
debug2: MACs stoc: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>
debug2: compression ctos: none,<email address hidden>
debug2: compression stoc: none,<email address hidden>
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-
debug2: host key algorithms: rsa-sha2-
debug2: ciphers ctos: <email address hidden>
debug2: ciphers stoc: <email address hidden>
debug2: MACs ctos: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>
debug2: MACs stoc: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>
debug2: compression ctos: none,<email address hidden>
debug2: compression stoc: none,<email address hidden>
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: (no match)
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
```
The error is most likely caused by a race condition on removing /run/sshd, which is easily reproducible by ssh-keyscan.
I noticed that depeding on client, I'd sometimes miss all keys, sometimes get one, sometimes more.
Modifying the following files (they should me marked as modified in the bug report) seems to solve the issue, at least temporarily:
/usr/lib/
/usr/lib/
In both cases, I added `RuntimeDirecto
This is the same solution mentioned in the Debian bug, although their bug report doesn't mention which service files are affected.
This doesn't seem to be a proper long-term solution though, as it seems apt doesn't respect configuration files in /usr (or they are unlisted somewhere),
because after upgrading system just before filing this bug report, the files got overwritten and reverted to their original form.
I only got asked about the /etc/ssh/
ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: openssh-server 1:8.2p1-4ubuntu0.1 [modified: lib/systemd/
ProcVersionSign
Uname: Linux 5.4.0-54-generic x86_64
ApportVersion: 2.20.11-
Architecture: amd64
CasperMD5CheckR
Date: Mon Nov 23 15:09:32 2020
SourcePackage: openssh
UpgradeStatus: No upgrade log present (probably fresh install)
Related branches
- Canonical Server packageset reviewers: Pending requested
- Canonical Server: Pending requested
-
Diff: 294 lines (+206/-17) (has conflicts)8 files modifieddebian/changelog (+41/-0)
debian/control (+9/-0)
debian/patches/CVE-2021-28041.patch (+14/-0)
debian/patches/lp-1876320-upstream-Do-not-call-process_queued_listen_addrs-for.patch (+59/-0)
debian/patches/lp1966591-upstream-preserve-group-world-read-permission-on-kno.patch (+46/-0)
debian/patches/match-host-certs-w-public-keys.patch (+30/-0)
debian/patches/series (+7/-0)
dev/null (+0/-17)
- Sergio Durigan Junior (community): Approve
- Canonical Server Core Reviewers: Pending requested
-
Diff: 26 lines (+9/-0)2 files modifieddebian/changelog (+8/-0)
debian/systemd/ssh@.service (+1/-0)
- Bryce Harrington (community): Approve
- Canonical Server: Pending requested
-
Diff: 26 lines (+9/-0)2 files modifieddebian/changelog (+8/-0)
debian/systemd/ssh@.service (+1/-0)
- Utkarsh Gupta (community): Approve
- Canonical Server: Pending requested
-
Diff: 26 lines (+9/-0)2 files modifieddebian/changelog (+8/-0)
debian/systemd/ssh@.service (+1/-0)
Changed in openssh (Ubuntu): | |
status: | New → Triaged |
importance: | Undecided → Medium |
description: | updated |
Changed in openssh (Ubuntu Focal): | |
assignee: | nobody → Athos Ribeiro (athos-ribeiro) |
Changed in openssh (Ubuntu Focal): | |
status: | New → In Progress |
Changed in openssh (Ubuntu Hirsute): | |
assignee: | nobody → Athos Ribeiro (athos-ribeiro) |
Hello Marcin, the Description section of https:/ /www.freedeskto p.org/software/ systemd/ man/systemd. unit.html gives information on how to modify configurations without having them undone by future updates; the systemctl edit command automates the process of using these local modifications.
Thanks