context deadline exceeded: unknown in containerd with latest runc version

Bug #1927219 reported by Colin GILLE
40
This bug affects 6 people
Affects Status Importance Assigned to Milestone
runc (Ubuntu)
Fix Released
High
Lucas Kanashiro
Bionic
Fix Released
High
Lucas Kanashiro
Focal
Fix Released
High
Lucas Kanashiro
Groovy
Fix Released
High
Lucas Kanashiro
Hirsute
Fix Released
High
Lucas Kanashiro

Bug Description

[Impact]

Several regressions were identified by upstream in version 1.0.0-rc93 and were fixed in version 1.0.0-rc94:

https://github.com/opencontainers/runc/releases/tag/v1.0.0-rc94

"This release fixes several regressions found in v1.0.0-rc93. We
recommend users update as soon as possible."

And in version 1.0.0-rc95 we also have the fix for CVE-2021-30465.

[Test Plan]

Per https://wiki.ubuntu.com/DockerUpdates, our test case is the autopkgtests

[Where problems could occur]

As usual, we deliver most benefit to our users by delivering an upstream experience. A risk of regressions is part of that.

[Original Message]

When upgrading runc to the latest version from focal-updates, Kubernetes, using containerd, fails to start new containers after a certain amount of container have been reached a certain amount (more than 100 ~ 150 containers).
With the previous version of runc, I was able to run more than 340 containers on a single server without any issue.

I got those logs in containerd (journalctl -u containerd)
```
May 05 00:48:17 node6 containerd[1368656]: time="2021-05-05T00:48:17.260349619Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
May 05 00:48:21 node6 containerd[1368656]: time="2021-05-05T00:48:21.807947009Z" level=error msg="get state for 71005cc34ac16f6976a7aebd72faa734792639b003c5dec67f666449cf6eb8d7" error="context deadline exceeded: unknown"
May 05 00:48:23 node6 containerd[1368656]: time="2021-05-05T00:48:23.810434820Z" level=error msg="get state for 71005cc34ac16f6976a7aebd72faa734792639b003c5dec67f666449cf6eb8d7" error="context deadline exceeded: unknown"
May 05 00:48:25 node6 containerd[1368656]: time="2021-05-05T00:48:25.813453574Z" level=error msg="get state for c6497bbf82f668f8b8b534204cd2efaa30a97f884da3dd5fca4f51489298f855" error="context deadline exceeded: unknown"
May 05 00:48:27 node6 containerd[1368656]: time="2021-05-05T00:48:27.816285268Z" level=error msg="get state for c6497bbf82f668f8b8b534204cd2efaa30a97f884da3dd5fca4f51489298f855" error="context deadline exceeded: unknown"
May 05 00:48:29 node6 containerd[1368656]: time="2021-05-05T00:48:29.823052797Z" level=error msg="get state for f90f054966ab6b78304797d420a1f948c85e5b6e425037417f8966c08cd9a0f1" error="context deadline exceeded: unknown"
May 05 00:48:31 node6 containerd[1368656]: time="2021-05-05T00:48:31.826412096Z" level=error msg="get state for f90f054966ab6b78304797d420a1f948c85e5b6e425037417f8966c08cd9a0f1" error="context deadline exceeded: unknown"
```

This version of runc triggered the problem:
```
runc (1.0.0~rc93-0ubuntu1~20.04.1) focal; urgency=medium

  * Backport version 1.0.0~rc93-0ubuntu1 from Hirsute (LP: #1919322,
    LP: #1916485).

 -- Lucas Kanashiro <email address hidden> Tue, 16 Mar 2021 15:34:35 -0300
```

```
# runc -v
runc version spec: 1.0.2-dev
go: go1.13.8
libseccomp: 2.5.1
```

Reverting to the previous version of runc solved the problem, and I was able to run more than 340 pods / containers without any error.
```
apt-get install runc=1.0.0~rc10-0ubuntu1

# runc -v
runc version spec: 1.0.1-dev
```

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: runc 1.0.0~rc93-0ubuntu1~20.04.1
ProcVersionSignature: Ubuntu 5.4.0-72.80-generic 5.4.101
Uname: Linux 5.4.0-72-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.16
Architecture: amd64
CasperMD5CheckResult: skip
Date: Wed May 5 12:06:30 2021
SourcePackage: runc
UpgradeStatus: No upgrade log present (probably fresh install)

CVE References

Revision history for this message
Colin GILLE (colingille) wrote :
Revision history for this message
Paride Legovini (paride) wrote :

Hello and thanks for this bug report. I had a look a the runc bug reports and changes to see if I could easily spot a relevant change, but I couldn't. I don't have a good hypothesis of what could be wrong, but I'd start checking if the newer runc is causing a higher memory or CPU usage, causing components to timeout ("context deadline exceeded") or OOMs.

Could you please:
 - Attach the kernel log (dmesg) to this bug, captured after
   hitting those containerd errors?
 - Check the system load and memory usage when using the two
   different versions of runc. This could be done for example
   via something like `vmstat -S M 5`.

If no useful clues are found, then I think we'll have to bisect.

Changed in runc (Ubuntu):
status: New → Incomplete
Revision history for this message
Colin GILLE (colingille) wrote :

Hi,
Here is the kern.log extract of the time containers were trying to start.
I only see logs about network interfaces, and the log is full of it.
I don't see those logs anymore after the runc version downgrade.

It looks like just a consequence of the runner's start failure.

About the load, I don't think it was very high, the server is quite powerful (256GB of RAM, 16 core / 32 threads).
I experienced the problem on a production system during what should have been a quick server reinstall during late maintenance hours, so I might be mistaken about the load at that time.

I'll try to reproduce the issue on a test environment this weekend to get real load figures with both versions of runc & allow further testing.

Revision history for this message
Colin GILLE (colingille) wrote :
Download full text (5.4 KiB)

I was able to reproduce the behaviour under a test VM.

I used a simple example nginx deployment with a high number of replicas.

With the older version of runc, I was able to run 200 replicas successfully and even 250.

With the newer version of runc, I was only able to run 164 replicas out of 200 (the last 36 pods were stuck in a ContainerCreating state.
There were also 2 system pods on the node (weave + kube-proxy), so a total of 166 containers.

```
nginx-deployment-66b6c48dd5-2hdgd 0/1 ContainerCreating 0 11m
nginx-deployment-66b6c48dd5-2ql8r 0/1 ContainerCreating 0 11m
nginx-deployment-66b6c48dd5-4z54r 0/1 ContainerCreating 0 11m
nginx-deployment-66b6c48dd5-5vvzl 0/1 ContainerCreating 0 11m
nginx-deployment-66b6c48dd5-6xh9g 0/1 ContainerCreating 0 11m
...
```

I saw the same logs as what I saw on the production server for containerd, and nothing more in dmesg / kern.log.

The load on the node was ok:
```
root@node0:~# vmstat -S M 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r b swpd free buff cache si so bi bo in cs us sy id wa st
 0 0 0 2305 70 997 0 0 140 390 1242 2560 5 4 92 0 0
 1 0 0 2305 70 998 0 0 0 0 2312 4529 1 1 98 0 0
 0 0 0 2303 70 998 0 0 0 3 5055 11301 4 4 92 0 0
 0 0 0 2303 70 998 0 0 0 0 2171 4232 1 1 98 0 0
 0 0 0 2302 70 998 0 0 0 0 5512 11719 4 4 92 0 0
 2 0 0 2302 70 998 0 0 0 0 2182 4261 1 1 98 0 0
 1 0 0 2302 70 998 0 0 0 0 3822 7883 3 1 96 0 0
 0 0 0 2302 70 998 0 0 0 0 3724 7891 3 3 95 0 0
 0 0 0 2303 70 998 0 0 0 0 2729 5351 3 1 96 0 0
 0 0 0 2300 70 998 0 0 0 4 4674 9863 3 3 94 0 0
 0 0 0 2300 70 998 0 0 0 0 2476 4930 2 1 97 0 0
 1 0 0 2305 70 997 0 0 0 65 5219 10770 4 4 91 0 0
 0 0 0 2430 70 994 0 0 0 2267 4918 9713 6 5 89 0 0
 0 0 0 2558 71 990 0 0 0 2033 7205 15041 7 7 85 1 0
 0 0 0 2589 71 989 0 0 0 1364 5006 10203 6 5 88 1 0
 0 0 0 2429 71 993 0 0 0 1554 9502 20112 12 13 74 1 0
 0 0 0 2325 71 997 0 0 0 1043 9185 19772 11 11 78 1 0
 0 0 0 2303 71 998 0 0 0 1810 3723 7264 4 4 91 0 0
 2 0 0 2318 71 998 0 0 0 0 3181 6494 3 2 96 0 0
 0 0 0 2317 71 998 0 0 0 23 2789 5525 2 1 97 0 0
 3 0 0 2307 71 999 0 0 26 84 5406 11430 5 4 90 0 0
 0 0 0 2307 71 999 0 0 0 31 2631 5201 2 1 97 0 0
 0 0 0 2306 71 999 0 0 0 5 2687 5532 2 1 97 0 0
 1 0 0 2306 71 999 0 0 0 221 4689 9833 3 3 94 0 0
 1 0 0 2305 71 999 0 0 0...

Read more...

Revision history for this message
Colin GILLE (colingille) wrote :
Download full text (3.2 KiB)

In order to test weave's container network impact, on the limit, I started 200 "sleep 7200" pods with the busybox image, with "hostNetwork: true" to disable container networking.

This time, I was able to run 200 replicas, but I was unable to go to over 200 pods, I tried the grow the replicas count to 300 and just got 100 pods in a ContainerCreating state, and the same logs.

```
piVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 200
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      hostNetwork: true
      containers:
      - name: nginx
        image: busybox
        command: ['sleep', '7200']
```

```
May 08 00:12:34 node0 containerd[399]: time="2021-05-08T00:12:34.406607997Z" level=warning msg="unknown status" status=0
May 08 00:12:36 node0 containerd[399]: time="2021-05-08T00:12:36.409698273Z" level=error msg="get state for 0c20bb658e597c7ef66089e910e525e4da03f4145f0dcee645202ae7bb18778c" error="context deadline exceeded: unknown"
May 08 00:12:36 node0 containerd[399]: time="2021-05-08T00:12:36.410297996Z" level=warning msg="unknown status" status=0
May 08 00:12:38 node0 containerd[399]: time="2021-05-08T00:12:38.413845203Z" level=error msg="get state for 04f407c1f0b373163b0f646351e9533b4219e4f33f7a0ddc6725314dce8a21e3" error="context deadline exceeded: unknown"
May 08 00:12:38 node0 containerd[399]: time="2021-05-08T00:12:38.413877394Z" level=warning msg="unknown status" status=0
May 08 00:12:40 node0 containerd[399]: time="2021-05-08T00:12:40.416039250Z" level=error msg="get state for e7a50022e2fb9806899f3d088b3d30562f012a0d550c7be19520b7bb8e8a6563" error="context deadline exceeded: unknown"
May 08 00:12:40 node0 containerd[399]: time="2021-05-08T00:12:40.416476408Z" level=warning msg="unknown status" status=0
```

```
May 08 00:14:43 node0 containerd[399]: time="2021-05-08T00:14:43.486466380Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/9e325f009e5a4456eea7f9037ef5e2513e8dd4634b0b4427b57faa8b367bc37e pid=132736
May 08 00:14:43 node0 containerd[399]: time="2021-05-08T00:14:43.518910596Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/dbfa2e9ebef9cc375080b05d32b19b5465f504cf102cea9c6b0ed1c07324bb9f pid=132762
May 08 00:14:43 node0 containerd[399]: time="2021-05-08T00:14:43.526959046Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/86a0e34b828552812d1e23da8445c20977c4d55383f3a4b30772297ed320da21 pid=132792
May 08 00:14:43 node0 containerd[399]: time="2021-05-08T00:14:43.555110616Z" level=info msg="shim disconnected" id=769c0f791b28650f343595fa1b2aef3605001ccbebcacbec8180a44a44a8565e
May 08 00:14:43 node0 containerd[399]: time="2021-05-08T00:14:43.568631491Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:nginx-deployment-5c4997c6fd-jk4nd,Uid:6588e20a-f4e4-46fa-9e0b-03a57fe4282b,Namespace:default,Attempt:0,} failed, error" error="failed to start sandbox container task \"769c0f791b28650f343595fa1b...

Read more...

Revision history for this message
Colin GILLE (colingille) wrote :

I tested the pre-built binary of the latest version of runc (by just replacing /usr/sbin/runc) and I was able to go to 400 pods on a single test node.
This new version was released today.

```
root@node0:~# runc -v
runc version 1.0.0-rc94
spec: 1.0.2-dev
go: go1.14.15
libseccomp: 2.5.1
```
https://github.com/opencontainers/runc/releases

I tried the pre-built binary of runc 1.0.0-rc93, and I was stuck to 164 pods again with "code = DeadlineExceeded desc = context deadline exceeded" error:
```
root@node0:~# runc -v
runc version 1.0.0-rc93
spec: 1.0.2-dev
go: go1.14.14
libseccomp: 2.5.1
```

The changelog for 1.0.0-rc94 contains this line that could match:
runc init: fix a hang caused by deadlock in seccomp/ebpf loading code
(regression in rc93, #2871)

The version 1.0.0-rc92 was working fine too.
```
root@node0:~# runc -v
runc version 1.0.0-rc92
spec: 1.0.2-dev
```

Please upgrade to version 1.0.0-rc94 or downgrade to 1.0.0-rc92.

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Thanks for the investigation! It seems that a bunch of regressions were found in 1.0.0~rc93 as we can see in the 1.0.0~rc94 release notes:

"This release fixes several regressions found in v1.0.0-rc93. We
recommend users update as soon as possible."

https://github.com/opencontainers/runc/releases/tag/v1.0.0-rc94

I'll be preparing a PPA with this new runc version and ask you to test it if possible.

Revision history for this message
Colin GILLE (colingille) wrote :

Of course, I could try a PPA or an attached deb file, anything that suits you best, everything is running on a test system anyway.
Thanks for your help!

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

I updated the runc package to version 1.0-rc94 and it is available in this PPA for testing:

https://launchpad.net/~lucaskanashiro/+archive/ubuntu/container-stack

Let me know if this indeed fix your issue. If yes, I'll update runc in the development release (a.k.a. impish) and then start the SRU process (I'll need you to perform the same tests at this stage).

Revision history for this message
Colin GILLE (colingille) wrote :

The version published in the PPA fixes the issue.

```
root@node0:~# runc -v
runc version spec: 1.0.2-dev
go: go1.13.8
libseccomp: 2.5.1
```

Fore some reason, the runc version published in the PPA don't show the actual runc version with `runc -v`, just like the previous version (1.0-rc93) published in the focal-update repository.
The patches applied to the orig source tree only relate to test fixes for the build environment.

Revision history for this message
Colin GILLE (colingille) wrote :

The missing version seems to be caused by a different build environment (missing VERSION file at the root of the source tree).
https://github.com/opencontainers/runc/blob/master/main.go#L20
https://github.com/opencontainers/runc/blob/master/Makefile#L15

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Thanks for raising this issue. The VERSION file is not missing but we are not using the upstream Makefile to build the source, that's why the version is not set.

I fixed this issue and uploaded the fix to my PPA (it might take some minutes to get it published), now if you update the runc package you will have the correct version when running 'runc --version'.

Revision history for this message
Paride Legovini (paride) wrote :

I'm marking this bug as Triaged: I didn't try to reproduce it locally, but I think it's clear enough we have a bug here. Also tagging this as a regression.

Changed in runc (Ubuntu):
status: Incomplete → Triaged
tags: added: regression-update server-next
Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

@Colin I updated my PPA with version 1.0-rc95 which also contains the fix for CVE-2021-30465. Could you test this new version and make sure your regression is fixed?

description: updated
Changed in runc (Ubuntu):
assignee: nobody → Lucas Kanashiro (lucaskanashiro)
Changed in runc (Ubuntu Focal):
assignee: nobody → Lucas Kanashiro (lucaskanashiro)
Changed in runc (Ubuntu Groovy):
assignee: nobody → Lucas Kanashiro (lucaskanashiro)
Changed in runc (Ubuntu Bionic):
assignee: nobody → Lucas Kanashiro (lucaskanashiro)
Changed in runc (Ubuntu Hirsute):
assignee: nobody → Lucas Kanashiro (lucaskanashiro)
Felipe Reyes (freyes)
tags: added: seg
Revision history for this message
Ante Karamatić (ivoks) wrote :

As Colin stated, this version of runc is causing a serious regression. As such it should be marked as High.

Changed in runc (Ubuntu):
importance: Undecided → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in runc (Ubuntu Bionic):
status: New → Confirmed
Changed in runc (Ubuntu Focal):
status: New → Confirmed
Changed in runc (Ubuntu Groovy):
status: New → Confirmed
Changed in runc (Ubuntu Hirsute):
status: New → Confirmed
Revision history for this message
Colin GILLE (colingille) wrote :

@Lucas, I just tested with runc 1.0.0~rc95-0ubuntu1~20.04.1~ppa1 from your PPA, everything is working ok (300 containers tested).

I don't know if the containerd update is planned, but I ran the test with containerd 1.3.3-0ubuntu2.3 from the Ubuntu repo and 1.5.2-0ubuntu1~20.04.1~ppa1 form your PPA, both were running ok.

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Thanks for the testing Colin!

FYI I already uploaded the fix to Impish (development release) and as soon it lands in the release pocket I can start the SRUs. At the moment it is blocked by one regression:

https://people.canonical.com/~ubuntu-archive/proposed-migration/impish/update_excuses.html#runc

I'll be working on that today to sort it out.

tags: added: sts
Dan Streetman (ddstreet)
Changed in runc (Ubuntu Bionic):
importance: Undecided → High
Changed in runc (Ubuntu Focal):
importance: Undecided → High
Changed in runc (Ubuntu Groovy):
importance: Undecided → High
Changed in runc (Ubuntu Hirsute):
importance: Undecided → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package runc - 1.0.0~rc95-0ubuntu1

---------------
runc (1.0.0~rc95-0ubuntu1) impish; urgency=medium

  * New upstream release.
    - Several regressions were found in 1.0.0-rc93 by upstream and fixed in
      this new release.
      + Ensure the scratch pipe is read during ExportBPF (LP: #1927219).
    - Drop patches applied by upstream:
      + d/patches/CVE-2021-30465/*.patch
      + d/patches/fix-patchpbf-test-on-32-bit.patch
  * d/rules: set VERSION variable when building runc (LP: #1929106).

 -- Lucas Kanashiro <email address hidden> Thu, 20 May 2021 10:40:14 -0300

Changed in runc (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Colin, or anyone else affected,

Accepted runc into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.0.0~rc95-0ubuntu1~21.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in runc (Ubuntu Hirsute):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-hirsute
Changed in runc (Ubuntu Groovy):
status: Confirmed → Fix Committed
tags: added: verification-needed-groovy
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Colin, or anyone else affected,

Accepted runc into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.0.0~rc95-0ubuntu1~20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Colin, or anyone else affected,

Accepted runc into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.0.0~rc95-0ubuntu1~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in runc (Ubuntu Focal):
status: Confirmed → Fix Committed
tags: added: verification-needed-focal
Changed in runc (Ubuntu Bionic):
status: Confirmed → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Colin, or anyone else affected,

Accepted runc into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.0.0~rc95-0ubuntu1~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Colin GILLE (colingille) wrote :

Hi Brian,

I tested the package from focal-proposed and everything is working as expected.

Tested version:
root@node0:~# runc -v
runc version 1.0.0~rc95-0ubuntu1~20.04.1
spec: 1.0.2-dev
go: go1.13.8
libseccomp: 2.5.1

Test:
Start 300 replicas of nginx on a single kuberentes node.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (runc/1.0.0~rc95-0ubuntu1~20.04.1)

All autopkgtests for the newly accepted runc (1.0.0~rc95-0ubuntu1~20.04.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

containerd/1.4.4-0ubuntu1~20.04.2 (s390x, ppc64el, arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#runc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (runc/1.0.0~rc95-0ubuntu1~21.04.1)

All autopkgtests for the newly accepted runc (1.0.0~rc95-0ubuntu1~21.04.1) for hirsute have finished running.
The following regressions have been reported in tests triggered by the package:

golang-github-containers-common/0.33.4+ds1-1 (s390x, arm64, ppc64el)
golang-github-docker-go-connections/0.4.0-2 (armhf, s390x, amd64, arm64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/hirsute/update_excuses.html#runc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (runc/1.0.0~rc95-0ubuntu1~18.04.1)

All autopkgtests for the newly accepted runc (1.0.0~rc95-0ubuntu1~18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

containerd/1.4.4-0ubuntu1~18.04.2 (ppc64el, s390x, i386, arm64)
containerd/unknown (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#runc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

I confirmed the golang-github-docker-go-connections regression in Hirsute and I filed another SRU bug to fix it:

https://bugs.launchpad.net/ubuntu/+source/golang-github-docker-go-connections/+bug/1930891

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

I've been trying to find a fix for the golang-github-containers-common/0.33.4+ds1-1 regression in Hirsute but without success so far. However, the issue seems to be unrelated to this runc SRU, it is a compilation issue in one of its components:

# github.com/containers/storage/pkg/devicemapper
../../../go-build3064824172/b263/_cgo_export.c:21:13: error: variable ‘_cgoexp_ebd00b0772db_StorageDevmapperLogCallback’ redeclared as function
   21 | extern void _cgoexp_ebd00b0772db_StorageDevmapperLogCallback(void *);
      | ^
../../../go-build3064824172/b263/cgo-generated-wrappers:1:5: note: previously declared here
lto1: fatal error: errors during merging of translation units
compilation terminated.

Due to that, I'd ask the SRU team to ignore this regression for now and not block this SRU just because of it.

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

The containerd regressions in Focal and Bionic were already sorted out, the tests are passing now.

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Regarding the golang-github-containers-common/0.33.4+ds1-1 regression in Hirsute, it was reported here in LP #1931258 as a LTO optimization issue. We could SRU this package adding the

export DEB_BUILD_MAINT_OPTIONS=optimize=-lto

to d/rules but I believe this does not worth a SRU. If the SRU team thinks this is important I can prepare an update with the content mentioned above (I confirmed it fixes the issue).

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :
Download full text (3.2 KiB)

I did the verification of the packages in -proposed in all releases:

Hirsute
-------

ubuntu@hirsute:~/runc/runc-1.0.0-rc95$ head -n 12 debian/changelog
runc (1.0.0~rc95-0ubuntu1~21.04.1) hirsute; urgency=medium

  * New upstream release.
    - Several regressions were found in 1.0.0-rc93 by upstream and fixed in
      this new release.
      + Ensure the scratch pipe is read during ExportBPF (LP: #1927219).
    - Drop patches applied by upstream:
      + d/patches/CVE-2021-30465/*.patch
      + d/patches/fix-patchpbf-test-on-32-bit.patch
  * d/rules: set VERSION variable when building runc (LP: #1929106).

 -- Lucas Kanashiro <email address hidden> Thu, 20 May 2021 10:53:22 -0300
ubuntu@hirsute:~/runc/runc-1.0.0-rc95$ sudo autopkgtest . -- null
...
autopkgtest [16:38:08]: @@@@@@@@@@@@@@@@@@@@ summary
basic-smoke PASS
command1 PASS

Groovy
------

ubuntu@groovy:~/runc/runc-1.0.0-rc95$ head -n 12 debian/changelog
runc (1.0.0~rc95-0ubuntu1~20.10.1) groovy; urgency=medium

  * New upstream release.
    - Several regressions were found in 1.0.0-rc93 by upstream and fixed in
      this new release.
      + Ensure the scratch pipe is read during ExportBPF (LP: #1927219).
    - Drop patches applied by upstream:
      + d/patches/CVE-2021-30465/*.patch
      + d/patches/fix-patchpbf-test-on-32-bit.patch
  * d/rules: set VERSION variable when building runc (LP: #1929106).

 -- Lucas Kanashiro <email address hidden> Thu, 20 May 2021 11:01:11 -0300
ubuntu@groovy:~/runc/runc-1.0.0-rc95$ sudo autopkgtest . -- null
...
autopkgtest [16:39:17]: @@@@@@@@@@@@@@@@@@@@ summary
basic-smoke PASS
command1 PASS

Focal
-----

ubuntu@focal:~/runc/runc-1.0.0-rc95$ head -n 12 debian/changelog
runc (1.0.0~rc95-0ubuntu1~20.04.1) focal; urgency=medium

  * New upstream release.
    - Several regressions were found in 1.0.0-rc93 by upstream and fixed in
      this new release.
      + Ensure the scratch pipe is read during ExportBPF (LP: #1927219).
    - Drop patches applied by upstream:
      + d/patches/CVE-2021-30465/*.patch
      + d/patches/fix-patchpbf-test-on-32-bit.patch
  * d/rules: set VERSION variable when building runc (LP: #1929106).

 -- Lucas Kanashiro <email address hidden> Thu, 20 May 2021 11:06:57 -0300
ubuntu@focal:~/runc/runc-1.0.0-rc95$ sudo autopkgtest . -- null
...
autopkgtest [16:41:19]: @@@@@@@@@@@@@@@@@@@@ summary
basic-smoke PASS
command1 PASS

Bionic
------

ubuntu@bionic:~/runc/runc-1.0.0-rc95$ head -n 12 debian/changelog
runc (1.0.0~rc95-0ubuntu1~18.04.1) bionic; urgency=medium

  * New upstream release.
    - Several regressions were found in 1.0.0-rc93 by upstream and fixed in
      this new release.
      + Ensure the scratch pipe is read during ExportBPF (LP: #1927219).
    - Drop patches applied by upstream:
      + d/patches/CVE-2021-30465/*.patch
      + d/patches/fix-patchpbf-test-on-32-bit.patch
  * d/rules: set VERSION variable when building runc (LP: #1929106).

 -- Lucas Kanashiro <email address hidden> Thu, 20 May 2021 11:11:34 -0300
ubuntu@bionic:~/runc/runc-1.0.0-rc95$ sudo autopkgtest . -- null
...
autopkgtest [16:42:02]: @@@@@@@@@@@@@@@@@@@@ summary
basic-sm...

Read more...

tags: added: verification-done verification-done-bionic verification-done-groovy verification-done-hirsute
removed: verification-needed verification-needed-bionic verification-needed-groovy verification-needed-hirsute
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package runc - 1.0.0~rc95-0ubuntu1~21.04.1

---------------
runc (1.0.0~rc95-0ubuntu1~21.04.1) hirsute; urgency=medium

  * New upstream release.
    - Several regressions were found in 1.0.0-rc93 by upstream and fixed in
      this new release.
      + Ensure the scratch pipe is read during ExportBPF (LP: #1927219).
    - Drop patches applied by upstream:
      + d/patches/CVE-2021-30465/*.patch
      + d/patches/fix-patchpbf-test-on-32-bit.patch
  * d/rules: set VERSION variable when building runc (LP: #1929106).

 -- Lucas Kanashiro <email address hidden> Thu, 20 May 2021 10:53:22 -0300

Changed in runc (Ubuntu Hirsute):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for runc has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package runc - 1.0.0~rc95-0ubuntu1~20.10.1

---------------
runc (1.0.0~rc95-0ubuntu1~20.10.1) groovy; urgency=medium

  * New upstream release.
    - Several regressions were found in 1.0.0-rc93 by upstream and fixed in
      this new release.
      + Ensure the scratch pipe is read during ExportBPF (LP: #1927219).
    - Drop patches applied by upstream:
      + d/patches/CVE-2021-30465/*.patch
      + d/patches/fix-patchpbf-test-on-32-bit.patch
  * d/rules: set VERSION variable when building runc (LP: #1929106).

 -- Lucas Kanashiro <email address hidden> Thu, 20 May 2021 11:01:11 -0300

Changed in runc (Ubuntu Groovy):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package runc - 1.0.0~rc95-0ubuntu1~18.04.1

---------------
runc (1.0.0~rc95-0ubuntu1~18.04.1) bionic; urgency=medium

  * New upstream release.
    - Several regressions were found in 1.0.0-rc93 by upstream and fixed in
      this new release.
      + Ensure the scratch pipe is read during ExportBPF (LP: #1927219).
    - Drop patches applied by upstream:
      + d/patches/CVE-2021-30465/*.patch
      + d/patches/fix-patchpbf-test-on-32-bit.patch
  * d/rules: set VERSION variable when building runc (LP: #1929106).

 -- Lucas Kanashiro <email address hidden> Thu, 20 May 2021 11:11:34 -0300

Changed in runc (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package runc - 1.0.0~rc95-0ubuntu1~20.04.1

---------------
runc (1.0.0~rc95-0ubuntu1~20.04.1) focal; urgency=medium

  * New upstream release.
    - Several regressions were found in 1.0.0-rc93 by upstream and fixed in
      this new release.
      + Ensure the scratch pipe is read during ExportBPF (LP: #1927219).
    - Drop patches applied by upstream:
      + d/patches/CVE-2021-30465/*.patch
      + d/patches/fix-patchpbf-test-on-32-bit.patch
  * d/rules: set VERSION variable when building runc (LP: #1929106).

 -- Lucas Kanashiro <email address hidden> Thu, 20 May 2021 11:06:57 -0300

Changed in runc (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.