iotop can raise UnicodeDecodeError because /proc/pid/status can have non-utf-8 data in Name

The attachment "fix-iotop-encoding-errors.diff" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Thank you for your bug report.

I can confirm this bug. Here are the steps to reproduce it:

# lxc launch images:ubuntu/focal iotop-bug1932523
# lxc shell iotop-bug1932523
$ apt update && apt install -y iotop gcc
$ echo -e '#include <unistd.h>\nint main(){sleep(1000);return 0;}' | gcc -x c -o 'Инстанц�' -
$ ./Инстанц�
## <Type Control-Z here to put the program in the backgroun>
$ iotop
Traceback (most recent call last):
  File "/usr/sbin/iotop", line 17, in <module>
    main()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 737, in main
    main_loop()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 727, in <lambda>
    main_loop = lambda: run_iotop(options)
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 620, in run_iotop
    return curses.wrapper(run_iotop_window, options)
  File "/usr/lib/python3.8/curses/__init__.py", line 105, in wrapper
    return func(stdscr, *args, **kwds)
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 612, in run_iotop_window
    ui.run()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 188, in run
    self.refresh_display(iterations == 0, total, current,
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 476, in refresh_display
    lines = self.get_data()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 457, in get_data
    return list(map(format, processes))
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 432, in format
    cmdline = p.get_cmdline()
  File "/usr/lib/python3/dist-packages/iotop/data.py", line 308, in get_cmdline
    proc_status = parse_proc_pid_status(self.pid)
  File "/usr/lib/python3/dist-packages/iotop/data.py", line 210, in parse_proc_pid_status
    for line in open('/proc/%d/status' % pid):
  File "/usr/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xef in position 20: invalid continuation byte

I was able to verify that this bug still happens on impish, using a recent version of iotop (0.6-24-g733f3f8-1.1).

@Mingun, I think the best course of action for this bug is to contact upstream and suggest the patch to them first. Woud you be willing to do that, please?

Thank you.

Hi folks,

Mingun: thanks for the patch, I would like to commit it upstream, but I
need to know what author name/email I should use for the commit and
what commit message to use.

If you know how to use git and already have a local commit, please use
the `git format-patch @{u}` command get a patch file with that info.

Once the commit author/message are available, I will commit the change
upstream and later I will update the Debian/Ubuntu package.

PS: until this gets fixed properly in Debian/Ubuntu, you might like to
use the iotop-c package from Debian/Ubuntu.

https://github.com/Tomas-M/iotop

--
bye,
pabs

https://bonedaddy.net/pabs3/

Hi Paul,

You can write any commit message that you find appropriate, I'm not familiar with common practices in the project and probably my English not so good. You also can made that commit without my attribution: this change is so simple that anyone can made it. I do not want to put progress of fixing bugs to the bureaucracy. But if you wish, you can use the following attributes:

name: Yanichkin Alexander
email: <email address hidden>

Thanks, I have forward-ported the patch, added a commit message and pushed the patch upstream:

https://repo.or.cz/iotop.git/commit/443737ec620a699286b9b2e44dbcaac53f553812

At some point in the next few weeks I'll make a new release, upload it to Debian and get it synced to Ubuntu.

Since the patch is only a workaround, it would be good to have a proper fix; guess the encoding using chardet or similar and coerce the bytes into Unicode. Any help with that is welcome.

 status fixcommitted

This has been fixed in 0.6-42-ga14256a-0.1, which is available from kinetic and on.

I am adding tasks for jammy, and focal on this one.

I assigned myself to Jammy and Focal. Btw, honestly, our "workaround patch" might be more safe in terms of regression possibilities, and it helps fix the problem with a minor change.

Hello Mingun, or anyone else affected,

Accepted iotop into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/iotop/0.6-24-g733f3f8-1.1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Mingun, or anyone else affected,

Accepted iotop into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/iotop/0.6-24-g733f3f8-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

The fix works, 0.6-24-g733f3f8-1.1ubuntu0.1 in Jammy fixes the bug.

I've created the jammy container using steps from the [Test Plan] section listed above in the Bug Description.

Then I installed iotop and gcc:

apt update && apt install -y iotop gcc

and inside that container I typed in:

$ apt policy iotop

The output:

iotop:
  Installed: 0.6-24-g733f3f8-1.1build2
  Candidate: 0.6-24-g733f3f8-1.1build2
  Version table:
 *** 0.6-24-g733f3f8-1.1build2 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status

Then I repeated the steps from the [Test Plan] section.

I've noticed that nothing has changed there, so the problem still exists:

Traceback (most recent call last):
  File "/usr/sbin/iotop", line 17, in <module>
    main()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 737, in main
    main_loop()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 727, in <lambda>
    main_loop = lambda: run_iotop(options)
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 620, in run_iotop
    return curses.wrapper(run_iotop_window, options)
  File "/usr/lib/python3.10/curses/__init__.py", line 94, in wrapper
    return func(stdscr, *args, **kwds)
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 612, in run_iotop_window
    ui.run()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 188, in run
    self.refresh_display(iterations == 0, total, current,
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 476, in refresh_display
    lines = self.get_data()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 457, in get_data
    return list(map(format, processes))
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 432, in format
    cmdline = p.get_cmdline()
  File "/usr/lib/python3/dist-packages/iotop/data.py", line 308, in get_cmdline
    proc_status = parse_proc_pid_status(self.pid)
  File "/usr/lib/python3/dist-packages/iotop/data.py", line 210, in parse_proc_pid_status
    for line in open('/proc/%d/status' % pid):
  File "/usr/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xef in position 20: invalid continuation byte

Then I've upgraded iotop using:
$ apt install iotop=0.6-24-g733f3f8-1.1ubuntu0.1

Later I've typed in:

$ apt policy iotop
to check if the installed version has changed, and we see that we have a new version installed (with a fix).

iotop:
  Installed: 0.6-24-g733f3f8-1.1ubuntu0.1
  Candidate: 0.6-24-g733f3f8-1.1ubuntu0.1
  Version table:
 *** 0.6-24-g733f3f8-1.1ubuntu0.1 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

Finally when I repeated steps from the [Test Plan] the problem did not exist, iotop works correctly, so the fix works.

-----

Focal - I performed an SRU Verification using the same steps as mentioned above and I can confirm that 0.6-24-g733f3f8-1ubuntu0.1 fixes the issue in Focal.

This bug was fixed in the package iotop - 0.6-24-g733f3f8-1.1ubuntu0.1

---------------
iotop (0.6-24-g733f3f8-1.1ubuntu0.1) jammy; urgency=medium

  * d/p/0001-Workaround-crashes-due-to-non-UTF-8-characters-in-pr.patch:
    Fix the crash caused by non-UTF-8 characters by using the Python
    feature of replacing non-UTF-characters with the U+FFFD replacement
    character. (LP: #1932523)

 -- Michal Maloszewski <email address hidden> Fri, 16 Jun 2023 15:55:00 +0200

The verification of the Stable Release Update for iotop has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package iotop - 0.6-24-g733f3f8-1ubuntu0.1

---------------
iotop (0.6-24-g733f3f8-1ubuntu0.1) focal; urgency=medium

  * d/p/crash-due-to-non-UTF-8-characters-in-process.patch: Fix the
    crash caused by non-UTF-8 characters by using the Python feature of
    replacing non-UTF-characters with the U+FFFD replacement character.
    (LP: #1932523)

 -- Michal Maloszewski <email address hidden> Mon, 19 Jun 2023 15:39:59 +0200