"User process fault: interruption code 0011 ilc:3" on SSH client/server upon Jammy upgrade

Bug #1970076 reported by Ryan Finnie
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
openssh (Ubuntu)
Expired
Undecided
Unassigned

Bug Description

This s390x VM has been upgraded to Jammy, including the host (qemu-system-s390x 1:6.2+dfsg-2ubuntu6). Now any attempt to use ssh, or any incoming sshd attempt immediately results in e.g.:

ryan@s390x-dev:~$ ssh <email address hidden>
[ 433.672800] User process fault: interruption code 0011 ilc:3 in libc.so.6[3ffa6c00000+1ca000]
[ 433.672946] Failing address: 000002aa0b84d000 TEID: 000002aa0b84d800
[ 433.672977] Fault in primary space mode while using user ASCE.
[ 433.673030] AS:0000000002bc01c7 R3:00000000087cc007 S:000000000798a000 P:0000000000000400
Segmentation fault (core dumped)

Revision history for this message
Ryan Finnie (fo0bar) wrote :
Ryan Finnie (fo0bar)
tags: added: apport-crash need-amd64-retrace
tags: added: need-s390x-retrace
removed: need-amd64-retrace
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi,
for comparison on another system I've taken a Host (was impish before upgrade) and created guests with Xenial, Bionic, Focal, Jammy.
All worked fine at this stage - ssh login and health of guests/hosts was good.
Then I upgraded the Host to Jammy (as the reporter did).

Example of the simple tests made:
#log into jammy, from there log into focal
ssh -t ubuntu@192.168.122.177 "ssh ubuntu@192.168.122.39"
#log into focal, from there log into jammy
ssh -t ubuntu@192.168.122.39 "ssh ubuntu@192.168.122.177"

=> That was all fine initially (impish)
=> That was all still fine after the upgrade to jammy
=> And it was still fine after restarting the guests (to use the new qemu)

So it can't be reproduced easily, this would need to go into analyzing the ssh crash dump file that is attached.

@Ryan - if you have any further insight on how/why your config might be special let us know.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

(gdb) bt
#0 __GI__IO_default_xsputn (n=<optimized out>, data=<optimized out>, f=<optimized out>) at libioP.h:947
#1 __GI__IO_default_xsputn (f=0x3ffcbc7c2e8, data=<optimized out>, n=2929360903402) at genops.c:370
#2 0x000003ffa6c7896c in outstring_func (done=11, length=2929360903402, string=0x2aa0b841cea <__func__.3.lto_priv.14> "read_config_file_depth", s=0x3ffcbc7c2e8) at ../libio/libioP.h:947
#3 __vfprintf_internal (s=s@entry=0x3ffcbc7c2e8, format=<optimized out>, format@entry=0x2aa0b81dea2 "%.48s:%.48s():%d (pid=%ld)", ap=ap@entry=0x3ffcbc7c4d0, mode_flags=mode_flags@entry=2)
    at vfprintf-internal.c:1517
#4 0x000003ffa6c8a024 in __vsnprintf_internal (string=0x3ffcbc7c5c8 "readconf.c:read_config_file_depth", maxlen=<optimized out>, format=0x2aa0b81dea2 "%.48s:%.48s():%d (pid=%ld)",
    args=args@entry=0x3ffcbc7c4d0, mode_flags=mode_flags@entry=2) at vsnprintf.c:114
#5 0x000003ffa6d2053a in ___snprintf_chk (s=<optimized out>, maxlen=<optimized out>, flag=<optimized out>, slen=<optimized out>, format=<optimized out>) at snprintf_chk.c:38
#6 0x000002aa0b7d1686 in snprintf (__fmt=0x2aa0b81dea2 "%.48s:%.48s():%d (pid=%ld)", __n=128, __s=0x3ffcbc7c5c8 "readconf.c:read_config_file_depth")
    at /usr/include/s390x-linux-gnu/bits/stdio2.h:71
#7 sshlogv (file=<optimized out>, func=<optimized out>, line=<optimized out>, showfunc=<optimized out>, level=<optimized out>, suffix=0x0,
    fmt=0x2aa0b84162e "Reading configuration data %.200s", args=0x3ffcbc7d3b0) at ../../log.c:473
#8 0x000002aa0b7d1b88 in sshlog (file=<optimized out>, func=<optimized out>, line=<optimized out>, showfunc=<optimized out>, level=SYSLOG_LEVEL_DEBUG1, suffix=0x0,
    fmt=0x2aa0b84162e "Reading configuration data %.200s") at ../../log.c:434
#9 0x000002aa0b80dbb6 in read_config_file_depth.constprop.0 (filename=0x2aa0b811b42 "/etc/ssh/ssh_config", pw=0x2aa0d3609d0, host=<optimized out>, original_host=<optimized out>,
    flags=<optimized out>, activep=0x3ffcbc7d64c, want_final_pass=0x3ffcbc7e7c4, depth=0, options=<optimized out>) at ../../readconf.c:2326
#10 0x000002aa0b793d62 in read_config_file (options=0x2aa0b8518e0 <options>, want_final_pass=0x3ffcbc7e7c4, flags=0, original_host=0x2aa0d364bc0 "github.com", host=<optimized out>,
    pw=0x2aa0d3609d0, filename=0x2aa0b811b42 "/etc/ssh/ssh_config") at ../../readconf.c:2295
#11 process_config_files (host_name=host_name@entry=0x2aa0d364bc0 "github.com", pw=pw@entry=0x2aa0d3609d0, final_pass=final_pass@entry=0, want_final_pass=want_final_pass@entry=0x3ffcbc7e7c4)
    at ../../ssh.c:569
#12 0x000002aa0b78d572 in main (ac=<optimized out>, av=<optimized out>) at ../../ssh.c:1148

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

@Ryan - Before going deeper since I read "readconf.c:read_config_file_depth" in there. Does the same happen on a fresh Jammy guest that has all-default config files? Or only to this particular guest that you have?

Revision history for this message
Ryan Finnie (fo0bar) wrote :

I only have one s390x guest and adding another would be painful due to the slowness of its emulation :) but I have just tried removing all of /etc/ssh and reinstalling openssh-client/openssh-server, same crash.

To be clear, this is only happening on my s390x guest. I have a few other non-x86 qemu guests (arm64, ppc64el, riscv64, etc) which have also been upgraded to jammy, none of which have this problem. So I'm willing to bet this is more of a kernel or qemu problem and that openssh is just a symptom. (Though it seems to be a recurring indicative symptom... ISTR openssh segfaults during the bootstrap of riscv64 on qemu; William Grant may be able to chime in.) I've attached my libvirt definition in case it's relevant, but it's pretty standard.

Revision history for this message
Jeff White (jjw867) wrote :

I can confirm that the issue may be in qemu in 22.04. A "real" s390x running 22.04 does not have this issue and the same installation image does not have this issue running on Windows 11, qemu version 7.0.0.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Oh I assumed this was running s390x VM on s390x Host.
@Ryan - is this s390x emulation on a non-s390x host?

Changed in openssh (Ubuntu):
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for openssh (Ubuntu) because there has been no activity for 60 days.]

Changed in openssh (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.