cgroups broken in impish

other information:-
container installed is from ubuntu template
can start/stop container
lxc-attach works

Hello and thanks for filing this bug report. I found this upstream issue:

https://github.com/lxc/lxcfs/issues/313

where Stéphane concludes that

> I expect the cgroup dir to be unavailable/broken on cgroup2 only
> systems but that shouldn't prevent you from using lxcfs as a kernel
> that's recent enough to provide cgroup2 is also recent enough to
> provide cgroup namespaces and so making that particular feature
> irrelevant.

so AIUI the "Input/output error" shouldn't be an issue.

On the lxc-console issue: I am not sure, but it doesn't look related to lxcfs. We need more details in order to confirm it's a bug in Ubuntu and to begin working on it. Could you please share some detailed steps to reproduce the issue from a clean Impish system (including the creation of the LXC container)?

While waiting for more information I'm marking this bug report as Incomplete.

Thanks!

to re-create the issue,
1 clean install of impish
# probably need sudo or be root for the following
2 boot into impish and install all updates
3 apt get install "lxc lxcfs lxc-utils lxc-templates lxctl"
4 lxc-create -n test -t ubuntu
5 lxc-start test
6 lxc-console -n test
# the above command FAILS to get a response, HOWEVER:-
7 lxc-console -t 0 -n test
# WORKS, only tty0 works in the container

==============================================================
THE CAUSE OF THE ISSUE!!
==============================================================

Previous to impish there are systemctl services to tend to all tty[1-4] and the console (tty0)
eg, on Focal Fossa, in the container :-
systemctl | grep tty # gives
  dev-lxc-tty1.mount loaded active mounted /dev/lxc/tty1
  dev-lxc-tty2.mount loaded active mounted /dev/lxc/tty2
  dev-lxc-tty3.mount loaded active mounted /dev/lxc/tty3
  dev-lxc-tty4.mount loaded active mounted /dev/lxc/tty4
  console-getty.service loaded active running Console Getty
  container-getty@0.service loaded active running Container Getty on /dev/pts/0
  container-getty@1.service loaded active running Container Getty on /dev/pts/1
  container-getty@2.service loaded active running Container Getty on /dev/pts/2
  container-getty@3.service loaded active running Container Getty on /dev/pts/3
  system-container\x2dgetty.slice loaded active active system-container\x2dgetty.slice
  system-getty.slice loaded active active system-getty.slice
  getty.target loaded active active Login Prompts

================================================================================
In impish, in the container,

systemctl | grep tty # gives
  dev-lxc-tty1.mount loaded active mounted /dev/lxc/tty1
  dev-lxc-tty2.mount loaded active mounted /dev/lxc/tty2
  dev-lxc-tty3.mount loaded active mounted /dev/lxc/tty3
  dev-lxc-tty4.mount loaded active mounted /dev/lxc/tty4
  console-getty.service loaded active running Console Getty
  system-getty.slice loaded active active system-getty.slice
  getty.target loaded active active Login Prompts

===============================================================================

As can be seen, there are no daemons started for any of the tty[1-4] consoles apart from console getty service.

The command lxc-console doesn't seem to default to tty0, so it picks a console that doesn't have a daemon to service it. Specifying tty0 (-t 0) makes it work. This seems to be a regre...

Just a thought, should we change the name of this report from cgroups to something "lxc-console only works for console tty0, not tty[1-4]" in impish?

Found the problem - for some reason the systemctl service getty@tty[1-4] has a prerequisite of /dev/tty0 existing. If it is not, all the tty[1-4] do not start. (tty1 fails, the others are not tried)
SEE below - "ConditionPathExists=/dev/tty0 was not met"

systemctl status getty@tty1
○ <email address hidden> - Getty on tty1
     Loaded: loaded (/lib/systemd/system/getty@.service; enabled; vendor preset>
     Active: inactive (dead)
  Condition: start condition failed at Wed 2021-10-20 21:50:42 AEDT; 4 days ago
             └─ ConditionPathExists=/dev/tty0 was not met
       Docs: man:agetty(8)
             man:systemd-getty-generator(8)
             http://0pointer.de/blog/projects/serial-console.html

If you create the file (eg sudo touch /etc/tty0 in the container) and then restart the tty daemons, all is good and it works.
eg
sudo systemctl restart getty@tty1
sudo systemctl restart getty@tty2
sudo systemctl restart getty@tty3
sudo systemctl restart getty@tty4

then all is good. However, this workaround needs to be done everytime the container is started - or you could remove the dependency on /dev/tty0 for the daemons.

Thanks for the instructions on how to reproduce your case. I did run containers in Focal and Impish in the way that you described and I confirm that the behavior is different between them. In Focal after bootstrapping and starting the container the lxc-console works as expected:

ubuntu@focal:~$ sudo lxc-console -n test

Connected to tty 1
Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself

Ubuntu 20.04.3 LTS test pts/0

test login: ubuntu
Password:
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-88-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@test:~$

But in Impish the same does not happen:

ubuntu@impish:~$ sudo lxc-console -n test

Connected to tty 1
Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself

Even if you try to pass another tty number there is no login prompt in the console, as you stated. I see you were checking for getty@.service but looking at the different services running inside both containers what I think we really want is container-getty@.service. The condition path in this service is /dev/pts/%I which is already satisfied in the Impish container when I logged in, all I had to do was starting this service and I was able to log in using the other tty numbers. We cannot enable this service by default because of its nature, so I am wondering if any of the conditions are not satisfied during the container boot time. In the syslog of the Focal container I can see:

Oct 26 19:48:17 test systemd[1]: Finished Permit User Sessions.
Oct 26 19:48:17 test systemd[1]: Started Console Getty.
Oct 26 19:48:17 test systemd[1]: Started Container Getty on /dev/pts/0.
Oct 26 19:48:17 test systemd[1]: Started Container Getty on /dev/pts/1.
Oct 26 19:48:17 test systemd[1]: Started Container Getty on /dev/pts/2.
Oct 26 19:48:17 test systemd[1]: Started Container Getty on /dev/pts/3.

And in the Impish one just this:

Oct 26 19:17:20 test systemd[1]: Finished Permit User Sessions.
Oct 26 19:17:20 test systemd[1]: Started Console Getty.

I did not have time to get to the bottom of this but it seems a real issue, I am marking this bug as Triaged and subscribing ubuntu-server for further investigation.

@stgraber - since this is lx[cd] and you still usually do the uploads. Do you have insight or opinion about this?

On Mon, Jan 03, 2022 at 02:44:10PM -0000, Christian Ehrhardt  wrote:
> @stgraber - since this is lx[cd] and you still usually do the uploads.
> Do you have insight or opinion about this?

LXCFS upstream contains a commit that will simply make
/var/lib/lxcfs/cgroup an empty directory without reporting any error.

Said change is in LXCFS 4.0.12 which is now in jammy