Can not ssh to github.com or gitlab.com when upgrading to 22.04

On Thu, May 05, 2022 at 09:09:07PM -0000, Alvaro wrote:
> acs@lsp-022:~$ ssh -vT <email address hidden>
> ...
> debug1: connect to address 140.82.121.4 port 22: Connection timed out

Note that "Connection timed out" is an error at the TCP level, that
indicates that your computer wasn't able to establish a TCP session. ssh's
algorithm choices aren't involved yet.

Are you sure this machine can communicate with 140.82.121.4:22 at all?

$ nc 140.82.112.4 22
SSH-2.0-babeld-78a8149e
^C

Thanks

Hi Seth,

Yes, I know it seems to be a problem with the TCP connection, but it is not:

```
acs@lsp-022:~$ nc github.com 22
SSH-2.0-babeld-78a8149e
```

```
acs@lsp-022:~$ nc gitlab.com 22
SSH-2.0-OpenSSH_8.4p1 Debian-5
```

```
acs@lsp-022:~$ ssh -Tv <email address hidden>
OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /home/acs/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to github.com [140.82.121.3] port 22.
debug1: connect to address 140.82.121.4 port 22: Connection timed out
```

This is why I have opened this bug, because it is really strange.

Maybe an incompatibility problem between ssh client and ssh server versions?

Cheers!

-- Alvaro

Hello Alvaro and thanks for this bug report. We have many systems running Jammy and they're able to connect to GitHub with no issues. I also tried with GitLab.com and it works just fine. This is not to dismiss your report, but there's clearly something else involved in the problem you're hitting. My suggestion is to try to reproduce the issue from a freshly installed Jammy system, and if ssh *does* work there go look for any relevant difference. I'm marking this report as Incomplete for now, as we need more information from your side to move it forward.

Hi Paride,

Ok, nice to hear that it is just an issue with my config.

I will try to isolate the issue and if the result is interesting, report it in this issue.

Thanks you for your awesome work with Ubuntu.

Cheers

-- Alvaro

Alvaro, I wonder if your network is dropping packets with unexpected IP QoS flags? Look for 'IPQoS' in ssh_config(5) to see the defaults and available choices. This would be influenced by ssh settings but still operate at TCP level.

Thanks

Morning Seth,

You hit the right key! Adding to ssh_config:

    IPQoS none

now it works correctly:

acs@lsp-022:~$ ssh -T <email address hidden>
Hi acs! You've successfully authenticated, but GitHub does not provide shell access.

With the default value (lowdelay) it just not connect.

With reliability it works, but with throughput, it does not.

It is strange that this is a problem with my network. It works in other laptops with previous versions of Ubuntu.

So my guess is that it is related just with the ssh client version.

If you need that I do more testing just ask for it.

Thank you very much.

-- Alvaro

Alvaro, thanks for reporting back! I'm glad it worked.

I don't know the full details of which QoS settings changed in which releases, but this email suggests that there was active interest in changing which exact values were used: http://lists.mindrot.org/pipermail/openssh-unix-dev/2018-April/036788.html

Thanks

Good catch, Seth :-).

Apparently this topic has been discussed in the past in bug #1822370 and https://lists.debian.org/debian-devel/2019/04/msg00010.html. Those discussions resulted in the decision to revert the IPQoS default values from openssh, which is something we're still doing. The default value for Debian/Ubuntu packages is "lowdelay", which, from what you said above, seems to be what's causing the problem for you.

This can be caused by how your router is handling QoS packets, for example. Either way, could you test this using a Debian testing container/VM and check if you can reproduce the issue there? If yes, then I would suggest opening a bug against the Debian openssh package, which is the best way to implement/discuss this decision IMHO. If you do so, please link the Debian bug here so that we can also keep track of its progress.

Thanks.

[Expired for openssh (Ubuntu) because there has been no activity for 60 days.]