IOTOP can not work due to "CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine SWAPIN and IO %"

Bug #1982727 reported by vodopad27
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
iotop (Ubuntu)
Fix Released
Low
Unassigned
Focal
Won't Fix
Low
Unassigned
Jammy
Won't Fix
Low
Unassigned
Kinetic
Fix Released
Low
Unassigned

Bug Description

[Impact]

* Users of iotop reported a crash on Jammy.

* This crash is caused by the task_delayacct value, which was not checked previously. The fix is to add a function with a try-catch block to the data.py file and moreover to alter some code in ui.py to detect the output of the try-catch block.

[Test Plan]

Make a container for testing:

$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test

Type in:

# apt install iotop

# iotop

Example of failed output:

The data under SWAPIN IO> column is:
?unavailable?

Moreover at the bottom of the screen there is a message:

CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine SWAPIN and IO %

Example of successful output:

There is a SWAPIN and IO % data that is readable.

[Where problems could occur]

* The patch itself modifies the format and refresh_display functions of the ui.py code, so the regressions should be limited to the behavior of data handling and display.

* Since the code changes affect task_delayacct, therefore potential regressions would most likely be related to other functionalities that make use of task_delayacct.

---------------------------original report--------------------------------------

iotop is not show disk usage due to error:
CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine SWAPIN and IO %.

Kernel params:
vodka@vodka-PC:~$ cat /boot/config-`uname -r` | grep CONFIG_TASK_DELAY_ACCT
CONFIG_TASK_DELAY_ACCT=y

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: iotop 0.6-24-g733f3f8-1.1build2
ProcVersionSignature: Ubuntu 5.15.0-41.44-generic 5.15.39
Uname: Linux 5.15.0-41-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
CasperMD5CheckResult: unknown
Date: Mon Jul 25 09:36:16 2022
Dependencies:

InstallationDate: Installed on 2018-05-02 (1544 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
SourcePackage: iotop
UpgradeStatus: Upgraded to jammy on 2022-04-27 (88 days ago)

Related branches

Revision history for this message
vodopad27 (family-gan) wrote :
Revision history for this message
Emanem (em4n3m) wrote (last edit ):

Hi, can confirm the same on Ubuntu 20.04 (kernel 5.15, HWE) - looks like the latest kernel has been built with this patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e4042ad492357fa995921376462b04a025dd53b6 (and perhaps in the past this was not the case?).

Hence by default one should add to grub the kernel option "delayacct" (as mentioned here: https://superuser.com/a/792913) as a workaround.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in iotop (Ubuntu):
status: New → Confirmed
Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Thanks for taking the time to report this bug and trying to make Ubuntu better.

This same bug was reported in Debian:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1004714

It should be fixed with this upstream commit:

https://repo.or.cz/iotop.git/commit/ab35334d374e588bec12d201fb8869c536f0545d

There are also a different workarounds than the one mentioned #2:

- sudo sysctl -w kernel.task_delayacct=1
- uninstall this package and install iotop-c instead

Changed in iotop (Ubuntu Focal):
importance: Undecided → Low
Changed in iotop (Ubuntu Jammy):
importance: Undecided → Low
Changed in iotop (Ubuntu Kinetic):
importance: Undecided → Low
tags: added: server-todo
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi,
I have looked at this package and it is neglected for too long.
Even being rather inactive upstream it was accumulating some bug fixes that would be quite useful to Ubuntu (and Debian as well).

I have converted this and other fixes into a debdiff as there is no Debian salsa repo nor is it imported into git-ubuntu yet.

I think it is time for an NMU and since we are before Ubuntu FF this will sync to here as well.
I'll ask a few fellow DDs to have a look at this debdiff that I attach here.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
tags: added: patch
Revision history for this message
Paride Legovini (paride) wrote :

Looks like this is one case where the official upstream tarball is not simply a git-archive(-like) generated tarball: the ChangeLog file your debdiff drops is not present in git, but is present in the upstream tarballs. This is a bit ugly, but upstream tags releases in git, and I personally prefer using upstream tags over tarballs when possible, so I'm +1 for the switch.

I am a bit surprised that there is no VCS for the Debian packaging, especially given that the Debian maintainer is an active DD and active user of salsa, and given that the upstream changes you are packaging are from him. He's listed in

  https://wiki.debian.org/LowThresholdNmu

so the NMU should be OK, but I would at least try to get in touch (maybe he has a packaging repo somewhere and just needs to push it somewhere).

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Yep, thank you (and Utkarsh) for your checks.
Adapted the minor things that came up in chat and otherwise ready now.

Didn't know about the LowThresholdNmu list, interesting TIL.

And I've already planned to ping Paul today and (due to his TZ) rather early tomorrow.
For now here an updated debdiff:

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi,
I've fixed this in Debian and it should over night sync into Ubuntu.
That will fix it in kinetic

For Focal - there the normal kernels aren't new enough to trigger the issue (>5.14).
So I doubt it is important to add noise there (happy to be convinced otherwise).

And finally in Jammy (and actually anything later as well) usually one should use iotop-c nowadays.

We have discussed removing the old iotop, but that will take a while still.
So not fixing it in jammy might help users to realize to better use iotop-c?
Not sure about that yet.
@vodopad27 - have you tried iotop-c in Jammy if that gives you what you need?

Changed in iotop (Ubuntu Kinetic):
status: Confirmed → In Progress
Changed in iotop (Ubuntu Jammy):
status: New → Incomplete
Changed in iotop (Ubuntu Focal):
status: New → Won't Fix
Revision history for this message
vodopad27 (family-gan) wrote :

Hello @paelzer

Yes, i tried to use iotop-c and it looks well. iotop-c works well and i see what i want.

There is one small disadvantage, i prefer "black" text and 'light' background, but in iotop-c utility "light" text and 'dark' background by default. But it is just small disadvantages of iotop-c in comparison with 'original' iotop.

Thanks, i will use iotop-c instead of iotop in the future.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package iotop - 0.6-42-ga14256a-0.1

---------------
iotop (0.6-42-ga14256a-0.1) unstable; urgency=medium

  * Non-maintainer upload.

  [ Christian Ehrhardt ]
  * New upstream release
    [still from git as there was no new tag since 9 years]
    - Among other fixes this new release includes various changes that
      fix some long waiting bugs:
      + 443737e Workaround crashes due to non-UTF-8 characters in
        process command-lines (Closes: #737043)
      + 0392b20 Ignore invalid lines in /proc/*/status files (Closes: #926207)
      + ab35334 Detect the kernel.task_delayacct sysctl value
        (Closes: #1004714)(LP: #1982727)

  [ Helmut Grohne ]
  * Fix Build-Depends for cross. (Closes: #913375)
    - Annotate python3 with :any.
    - Replace python3:any with python3-all:any as pybuild insists.
    - Add libpython3-all-dev as pybuild needs that as well.

 -- Christian Ehrhardt <email address hidden> Thu, 04 Aug 2022 09:39:29 +0200

Changed in iotop (Ubuntu Kinetic):
status: In Progress → Fix Released
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Full upgrade for Kinetic worked \o/

For the SRU we might isolate a few individual patches for which we have clear reproducers.

tags: added: bitesize
Changed in iotop (Ubuntu Jammy):
assignee: nobody → Michał Małoszewski (michal-maloszewski99)
Revision history for this message
Michał Małoszewski (michal-maloszewski99) wrote :

Taking care of that bug from 9th of August

Changed in iotop (Ubuntu Jammy):
assignee: Michał Małoszewski (michal-maloszewski99) → nobody
tags: removed: server-todo
Revision history for this message
Ken Sharp (kennybobs) wrote :

iotop-c -oPa sort of works, but there's a message about task_delayacct being off, and pressing CTRL+T enables it. But when doing so no processes are printed at all. This is both useless and not expected behaviour (compared to "old" iotop).

iotop-c is not a replacement for iotop.

Changed in iotop (Ubuntu Jammy):
status: Incomplete → Confirmed
Revision history for this message
Ken Sharp (kennybobs) wrote :

Okay, so... I was a bit too quick. It does show processes... eventually. I don't know why it takes so long but my guess would be the "delayacct" part.

tags: removed: bitesize
Revision history for this message
Michał Małoszewski (michal-maloszewski99) wrote (last edit ):

I did test the bahavior and https://repo.or.cz/iotop.git/commit/ab35334d374e588bec12d201fb8869c536f0545d does not fix that problem at all in Jammy.

description: updated
description: updated
Changed in iotop (Ubuntu Jammy):
assignee: nobody → Michał Małoszewski (michal-maloszewski99)
status: Confirmed → In Progress
description: updated
description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi everyone, sorry - I think I need to de-ambiguize a few things here.

Let me give the involved configs names:
A) CONFIG_TASK_DELAY_ACCT = yes, delayacct = 0
B) CONFIG_TASK_DELAY_ACCT = yes, delayacct = 1
C) CONFIG_TASK_DELAY_ACCT = no

#1 intentional config change
a) Past kernels had:
  - CONFIG_TASK_DELAY_ACCT enabled (which means the kernel CAN account that)
  - delayacct was enabled by default
  - so they were case (B) by default
  - So iotop worked out of the box to show e.g. swapin

b) newer kernels (post [1]) had:
  - CONFIG_TASK_DELAY_ACCT enabled (which means the kernel CAN account that)
  - delayacct was DISABLED by default
  - so they were case (A) by default
  - so iotop was unable to show swapin out of the box
  - It hinted at it and said "CONFIG_TASK_DELAY_ACCT and kernel.task_delayacct sysctl not enabled in kernel, cannot determine SWAPIN and IO %"

Now this is an intentional change in the kernel it saves some cpu and memory (not much but it sums up) for something not used in most cases.

Actually iotop is correctly detecting this situation of missing the stats (so what should it show) and hints at the problem.
I agree, in a perfect world it should instead hint at "sudo sysctl -w kernel.task_delayacct=1" or the "delayacct" kernel parameter, because as mentioned above CONFIG_TASK_DELAY_ACCT is enabled in kernel config.

The confusion starts at the fixes that are in upstream and the communication about it.
They are -not- changing the situation with (B).
There is no accounting done, it can not display anything.

What it does fix is that (C) (which aren't any Ubuntu kernels btw) that can therefore not read /proc/sys/kernel/task_delayacct do no more crash. Instead - with the fix - (C) will behave like (A).

So if your problem was:
- (C) is crashing -> that is fixed here
  - but since it isn't a situation you can get with Ubuntu kernels AFAIK this won't be SRUable
  - setting won't fix due to that for the SRU tasks
- (A) why don't I see results, then you need to change the runtime config
   - that is just a choice in the new kernel (since the default safes mem&cpu)
   - boot with delayacct
   - or run "sudo sysctl -w kernel.task_delayacct=1"
   - then you'll be in (B) and it works
- (B) just worked before and after

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e4042ad492357fa995921376462b04a025dd53b6

Changed in iotop (Ubuntu Jammy):
status: In Progress → Won't Fix
assignee: Michał Małoszewski (michal-maloszewski99) → nobody
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI I added a section to the jammy release notes to help less people to stumble over that (or if they do at least find an answer why and how to overcome). That is often higher in search rankings than an individual bug.

Revision history for this message
Ken Sharp (kennybobs) wrote :

   - or run "sudo sysctl -w kernel.task_delayacct=1"
   - then you'll be in (B) and it works

Tested just to confirm: this does work but it doesn't _seem_ to work straight away (because there's a delay I guess).

If you need this you can add a drop-in to your /etc/systctl.d/.

Thanks, Christian!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.