lxc-start: Error creating cgroups

Bug #1176287 reported by Reinhard Tartler
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
lxc (Ubuntu)
Fix Released
Critical
Serge Hallyn
Raring
Fix Released
Critical
Unassigned
Saucy
Fix Released
Critical
Serge Hallyn

Bug Description

=========== SRU Justification ===========
1. Impact: users who have cgroups mounted in more than one place will hang thier
system when starting an lxc container
2. Development fix: check for duplicate cgroup mounts
3. Stable fix: same as development fix
4. Test case:
sudo apt-get install lxc
sudo lxc-create -t ubuntu -n r1
sudo mkdir /var/cgroups
for d in `sed -e '1d;s/\([^\t]\)\t.*$/\1/' /proc/cgroups`; do
        sudo mkdir -p /var/cgroups/$d
        sudo mount -t cgroup -o $d cgroup /var/cgroups/$d
done
sudo lxc-start -n r1

If the lxc-start does not hang, the bug is not present.
5. Regression potential:

=========== SRU Justification ===========

I've played around with steam-lxc, but the container fails to start. Some debugging reveals:

>> sudo lxc-start -l DEBUG --name siretart-steam
lxc-start: Error creating cgroups
lxc-start: failed to spawn 'siretart-steam'

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: lxc 0.9.0-0ubuntu3
ProcVersionSignature: Ubuntu 3.8.0-19.30-generic 3.8.8
Uname: Linux 3.8.0-19-generic x86_64
NonfreeKernelModules: fglrx
ApportVersion: 2.9.2-0ubuntu8
Architecture: amd64
Date: Sat May 4 08:47:37 2013
InstallationDate: Installed on 2011-04-21 (743 days ago)
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Release amd64 (20101007)
MarkForUpload: True
SourcePackage: lxc
UpgradeStatus: Upgraded to raring on 2013-04-24 (9 days ago)

Revision history for this message
Reinhard Tartler (siretart) wrote :
Revision history for this message
Reinhard Tartler (siretart) wrote :

From: /var/log/lxc/siretart-steam.log

      lxc-start 1367650042.758 INFO lxc_start_ui - using rcfile /var/lib/lxc/siretart-steam/config
      lxc-start 1367650042.758 INFO lxc_apparmor - aa_enabled set to 1

      lxc-start 1367650042.758 DEBUG lxc_start - sigchild handler set
      lxc-start 1367650042.758 INFO lxc_start - 'siretart-steam' is initialized
      lxc-start 1367650042.795 DEBUG lxc_start - Not dropping cap_sys_boot or watching utmp

      lxc-start 1367650042.796 DEBUG lxc_conf - mac address of host interface 'veth4gEyxz' changed to private fe:fe:c5:7b:d3:72
      lxc-start 1367650042.796 DEBUG lxc_conf - instanciated veth 'veth4gEyxz/vethieShz2', index is '20'
      lxc-start 1367650046.208 ERROR lxc_cgroup - Error creating cgroups
      lxc-start 1367650046.298 ERROR lxc_start - failed to spawn 'siretart-steam'

Revision history for this message
Reinhard Tartler (siretart) wrote :
Revision history for this message
Reinhard Tartler (siretart) wrote :

How can I debug the problem further?

Revision history for this message
Reinhard Tartler (siretart) wrote :

the lxc-start command seems to massively create cgroups:

>> find /sys/fs/cgroup/ -name 'siretart-steam*' -print | wc -l
589773

What does create that many cgroups?

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Coudl you please show the result of

find /sys/fs/cgroup -maxdepth 3

Also, please post the full debug.out file after doing

sudo lxc-start -l info -o debug.out --name siretart-steam

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Sorry, please also show the output of:

dpkg -l | grep cgroup
cat /proc/self/mounts

Changed in lxc (Ubuntu):
status: New → Incomplete
importance: Undecided → High
Revision history for this message
Reinhard Tartler (siretart) wrote :
Revision history for this message
Reinhard Tartler (siretart) wrote :
Revision history for this message
Reinhard Tartler (siretart) wrote :
Revision history for this message
Reinhard Tartler (siretart) wrote :
Revision history for this message
Reinhard Tartler (siretart) wrote :

I've added all requested information now. Please indicate if there is anything else that may be helpful for diagnosing the problem.

Changed in lxc (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks for the information. The key was in the /proc/self/mounts output: you have cgroups mounted in schroots. This is triggering a bug in the way cgroups are handled in lxc.

You should be able to work around it by unmounting the cgroup filesystems from under /var/lib/schroot/mount/rtmpdump-test/sys/fs/cgroup.

I'll work on a fix for lxc.

Changed in lxc (Ubuntu):
assignee: nobody → Serge Hallyn (serge-hallyn)
importance: High → Critical
status: Confirmed → In Progress
Changed in lxc (Ubuntu Raring):
importance: Undecided → Critical
status: New → In Progress
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxc - 0.9.0-0ubuntu7

---------------
lxc (0.9.0-0ubuntu7) saucy; urgency=low

  * 0005-cgroup-prevent-DOS-when-a-hierachy-is-mounted-multip.patch: prevent
    DOS when a cgroup hierarchy is mounted multiple times (LP: #1176287)
 -- Serge Hallyn <email address hidden> Wed, 15 May 2013 22:19:59 +0000

Changed in lxc (Ubuntu Saucy):
status: In Progress → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Reinhard, or anyone else affected,

Accepted lxc into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/lxc/0.9.0-0ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in lxc (Ubuntu Raring):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Reinhard Tartler (siretart) wrote :

This new version of lxc does indeed fix the symptoms for me on raring. Thanks for the prompt update.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Adam Conrad (adconrad) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxc - 0.9.0-0ubuntu3.2

---------------
lxc (0.9.0-0ubuntu3.2) raring-proposed; urgency=low

  * 0004-cgroup-prevent-DOS-when-a-hierachy-is-mounted-multip.patch: prevent
    DOS when a cgroup hierarchy is mounted multiple times (LP: #1176287)
 -- Serge Hallyn <email address hidden> Wed, 15 May 2013 22:44:11 +0000

Changed in lxc (Ubuntu Raring):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.