Ubuntu
ubuntu-release-upgrader package

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Bug #1985964 reported by Alistair Buxton on 2022-08-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	ubuntu-release-upgrader (Ubuntu)	Fix Released	Medium	Unassigned
	Jammy	Fix Released	Medium	Nick Rosbrook

Bug Description

[Impact]

Crashes in ubuntu-release-upgrader cannot be properly reported with apport.

[Test plan]

Seeing this issue requires causing another exception to be raised first. I do this like so:

* Start an upgrade, but say no when prompted to start the upgrade, after the tarball has been downloaded.
* Navigate to the tempdir where the tarball was unpacked, e.g. /tmp/ubuntu-release-upgrader-04y_8xfj
* Edit DistUpgradeViewText.py, and add a divide-by-zero in `confirmChanges`
* Observe that instead of an apport crash report being generated, there is an exception trying to do so:

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/problem_report.py", line 477, in add_to_existing
    self.write(f)
  File "/usr/lib/python3/dist-packages/problem_report.py", line 430, in write
    block = f.read(1048576)
  File "/usr/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

[Where problems could occur]
The patch makes it so that u-r-u passes filenames, instead of open files, to apport to generate the report. Regressions would be seen in the generation of reports, or the format of the resulting report.

[Original Description]

While attempting to do-release-upgrade from 20.04 to 22.04, the upgrader crashed, and then the problem report script also crashed, leaving behind a corrupted crash report in /var/crash. This bug is about the problem report crash. I will report the upgrader crash separately.

Here are the last few lines of output:

Running mktexlsr /var/lib/texmf ... done.
Building format(s) --all.
This may take some time... done.
Processing triggers for dbus (1.12.20-2ubuntu4) ...
Processing triggers for libvlc-bin:amd64 (3.0.16-1build7) ...
Processing triggers for libgdk-pixbuf-2.0-0:amd64 (2.42.8+dfsg-1) ...
Processing triggers for libgdk-pixbuf-2.0-0:i386 (2.42.8+dfsg-1) ...
Processing triggers for rygel (0.40.3-1ubuntu2) ...
Reading package lists... Done
Building dependency tree
Reading state information... Done

Calculating the changes

Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-ksd7z8sv/jammy", line 8, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-ksd7z8sv/DistUpgrade/DistUpgradeMain.py", line 241, in main
    if app.run():
  File "/tmp/ubuntu-release-upgrader-ksd7z8sv/DistUpgrade/DistUpgradeController.py", line 2042, in run
    return self.fullUpgrade()
  File "/tmp/ubuntu-release-upgrader-ksd7z8sv/DistUpgrade/DistUpgradeController.py", line 1991, in fullUpgrade
    if not self.calcDistUpgrade():
  File "/tmp/ubuntu-release-upgrader-ksd7z8sv/DistUpgrade/DistUpgradeController.py", line 1100, in calcDistUpgrade
    if not self.cache.installTasks(self.tasks):
AttributeError: 'DistUpgradeController' object has no attribute 'tasks'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/problem_report.py", line 477, in add_to_existing
    self.write(f)
  File "/usr/lib/python3/dist-packages/problem_report.py", line 430, in write
    block = f.read(1048576)
  File "/usr/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Original exception was:
Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-ksd7z8sv/jammy", line 8, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-ksd7z8sv/DistUpgrade/DistUpgradeMain.py", line 241, in main
    if app.run():
  File "/tmp/ubuntu-release-upgrader-ksd7z8sv/DistUpgrade/DistUpgradeController.py", line 2042, in run
    return self.fullUpgrade()
  File "/tmp/ubuntu-release-upgrader-ksd7z8sv/DistUpgrade/DistUpgradeController.py", line 1991, in fullUpgrade
    if not self.calcDistUpgrade():
  File "/tmp/ubuntu-release-upgrader-ksd7z8sv/DistUpgrade/DistUpgradeController.py", line 1100, in calcDistUpgrade
    if not self.cache.installTasks(self.tasks):
AttributeError: 'DistUpgradeController' object has no attribute 'tasks'

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: apport 2.20.11-0ubuntu27.24
ProcVersionSignature: Ubuntu 5.4.0-124.140-generic 5.4.195
Uname: Linux 5.4.0-124-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.11-0ubuntu27.24
Architecture: amd64
CasperMD5CheckResult: skip
CrashReports:
644:0:119:0:2022-08-12 11:36:12.283871223 +0100:2022-08-12 11:36:12.283871223 +0100:/var/crash/_usr_bin_do-release-upgrade.0.upload
640:0:119:15237203:2022-08-12 11:36:10.283871173 +0100:2022-08-12 11:36:44.453251015 +0100:/var/crash/_usr_bin_do-release-upgrade.0.crash
600:112:119:5:2022-08-12 11:36:12.355865391 +0100:2022-08-12 11:36:12.355865391 +0100:/var/crash/_usr_bin_do-release-upgrade.0.uploaded
640:1000:119:2093621:2022-08-12 11:33:46.059253454 +0100:2022-08-12 11:33:47.059253454 +0100:/var/crash/_usr_lib_x86_64-linux-gnu_xfce4_panel_wrapper-2.0.1000.crash
CurrentDesktop: XFCE
Date: Fri Aug 12 11:37:21 2022
InstallationDate: Installed on 2018-05-07 (1557 days ago)
InstallationMedia: Xubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
PackageArchitecture: all
SourcePackage: apport
UpgradeStatus: Upgraded to focal on 2022-08-12 (0 days ago)

See original description

Tags:

Related branches

~enr0n/ubuntu-release-upgrader:ubuntu/jammy

Merged into ubuntu-release-upgrader:ubuntu/jammy at revision 8314486478db042d1eaf5d03ad2fc8ff5ec1d56b

Brian Murray: Approve on 2023-03-01

~enr0n/ubuntu-release-upgrader:lp.1991096

Merged into ubuntu-release-upgrader:ubuntu/main at revision 7c76ad3085c1a319e15bde38740018e28da44ad1

Brian Murray: Approve on 2022-09-30

Nick Rosbrook: Needs Fixing on 2022-09-29

Revision history for this message

Alistair Buxton (a-j-buxton) wrote on 2022-08-12:

ApportLog.txt Edit (1.8 KiB, text/plain; charset="utf-8")
Dependencies.txt Edit (3.9 KiB, text/plain; charset="utf-8")
ProcCpuinfoMinimal.txt Edit (1.3 KiB, text/plain; charset="utf-8")
ProcEnviron.txt Edit (112 bytes, text/plain; charset="utf-8")

Revision history for this message

Brian Murray (brian-murray) wrote on 2022-08-12:

The issue with ubuntu-release-upgrader is being tracked in bug 1969786.

no longer affects:

ubuntu-release-upgrader (Ubuntu)

Revision history for this message

Alistair Buxton (a-j-buxton) wrote on 2022-08-14:

The full upgrade logs are available on bug 1985966 which I have marked as a duplicate of bug 1969786.

Revision history for this message

Nick Rosbrook (enr0n) wrote on 2022-09-29:

Upon further investigation, the apport exception is caused by ubuntu-release-upgrader not passing open files to apport correctly.

affects:	apport (Ubuntu) → ubuntu-release-upgrader (Ubuntu)
Changed in ubuntu-release-upgrader (Ubuntu):
importance:	Undecided → Medium
status:	New → Triaged

Revision history for this message

Launchpad Janitor (janitor) wrote on 2022-10-08:

This bug was fixed in the package ubuntu-release-upgrader - 1:22.10.8

---------------
ubuntu-release-upgrader (1:22.10.8) kinetic; urgency=medium

  * DistUpgrade: Cast to int when calling QProgressBar.setValue() (LP: #1991096)
  * DistUpgrade: Just pass filename to apport report (LP: #1985964)
  * DistUpgrade: Remove firefox from deb2snap.json (LP: #1964036)
  * Run pre-build.sh: updating mirrors, demotions, and translations.

-- Nick Rosbrook <email address hidden> Thu, 29 Sep 2022 15:13:49 -0400

Changed in ubuntu-release-upgrader (Ubuntu):
status:	Triaged → Fix Released

Nick Rosbrook (enr0n) on 2022-10-13

Changed in ubuntu-release-upgrader (Ubuntu Jammy):
importance:	Undecided → Medium
status:	New → Triaged

Nick Rosbrook (enr0n) on 2022-10-13

description:

updated

Brian Murray (brian-murray) on 2022-11-15

Changed in ubuntu-release-upgrader (Ubuntu Jammy):
status:	Triaged → In Progress
assignee:	nobody → Nick Rosbrook (enr0n)

Revision history for this message

Brian Murray (brian-murray) wrote on 2022-11-29: Please test proposed package

Hello Alistair, or anyone else affected,

Accepted ubuntu-release-upgrader into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:22.04.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ubuntu-release-upgrader (Ubuntu Jammy):
status:	In Progress → Fix Committed
tags:	added: verification-needed verification-needed-jammy

Revision history for this message

Ubuntu SRU Bot (ubuntu-sru-bot) wrote on 2022-11-29: Autopkgtest regression report (ubuntu-release-upgrader/1:22.04.15)

All autopkgtests for the newly accepted ubuntu-release-upgrader (1:22.04.15) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-release-upgrader/1:22.04.15 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#ubuntu-release-upgrader

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message

Nick Rosbrook (enr0n) wrote on 2022-12-05:

Download full text (3.6 KiB)

The autopkgtest failure was resolved with a retry.

I verified the fix using ubuntu-release-upgrader 1:22.04.15 from jammy-proposed:

[ ... ]

Continue [yN] Details [d]N

Restoring original system state

Aborting
Reading package lists... Done
Building dependency tree
Reading state information... Done
nr@clean-focal-amd64:~$ sudo vim /tmp/ubuntu-release-upgrader-wah0wx3w/DistUpgradeViewText.py
nr@clean-focal-amd64:~$ grep -A 10 'def confirmChanges' /tmp/ubuntu-release-upgrader-wah0wx3w/DistUpgradeViewText.py
    def confirmChanges(self, summary, changes, demotions, downloadSize,
                       actions=None, removal_bold=True):
      DistUpgradeView.confirmChanges(self, summary, changes, demotions,
                                     downloadSize, actions)
      print(2/0) # Cause an exception
      print()
      print(twrap(summary))
      print(twrap(self.confirmChangesMessage))
      print(" %s %s" % (_("Continue [yN] "), _("Details [d]")), end="")
      while True:
        res = readline().strip().lower()
nr@clean-focal-amd64:~$ cd /tmp/ubuntu-release-upgrader-wah0wx3w/
nr@clean-focal-amd64:/tmp/ubuntu-release-upgrader-wah0wx3w$ sudo -E ./jammy --frontend=DistUpgradeViewText

Reading cache

Checking package manager
Reading package lists... Done
Building dependency tree
Reading state information... Done
Hit http://archive.ubuntu.com/ubuntu focal InRelease
Hit http://security.ubuntu.com/ubuntu focal-security InRelease
Hit http://archive.ubuntu.com/ubuntu focal-updates InRelease
Fetched 0 B in 0s (0 B/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
None

Checking for installed snaps

Calculating snap size requirements

Updating repository information
Hit http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit http://archive.ubuntu.com/ubuntu jammy InRelease
Hit http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Fetched 0 B in 0s (0 B/s)

Checking package manager
Reading package lists... Done
Building dependency tree
Reading state information... Done

Calculating the changes

The autopkgtest failure was resolved with a retry.

I verified the fix using ubuntu-release-upgrader 1:22.04.15 from jammy-proposed:

nr@clean-focal-amd64:~$ do-release-upgrade --proposed
Checking for a new Ubuntu release
Get:1 Upgrade tool signature [819 B]                                           
Get:2 Upgrade tool [1,267 kB]                                                  
Fetched 1,267 kB in 0s (0 B/s)                                                 
authenticate 'jammy.tar.gz' against 'jammy.tar.gz.gpg' 
extracting 'jammy.tar.gz'

[ ... ]

Continue [yN]  Details [d]N

Restoring original system state

Aborting
Reading package lists... Done    
Building dependency tree          
Reading state information... Done
nr@clean-focal-amd64:~$ sudo vim /tmp/ubuntu-release-upgrader-wah0wx3w/DistUpgradeViewText.py 
nr@clean-focal-amd64:~$ grep -A 10 'def confirmChanges' /tmp/ubuntu-release-upgrader-wah0wx3w/DistUpgradeViewText.py 
    def confirmChanges(self, summary, changes, demotions, downloadSize,
                       actions=None, removal_bold=True):
      DistUpgradeView.confirmChanges(self, summary, changes, demotions, 
                                     downloadSize, actions)
      print(2/0) # Cause an exception
      print()
      print(twrap(summary))
      print(twrap(self.confirmChangesMessage))
      print(" %s %s" % (_("Continue [yN] "), _("Details [d]")), end="")
      while True:
        res = readline().strip().lower()
nr@clean-focal-amd64:~$ cd /tmp/ubuntu-release-upgrader-wah0wx3w/
nr@clean-focal-amd64:/tmp/ubuntu-release-upgrader-wah0wx3w$ sudo -E ./jammy --frontend=DistUpgradeViewText

Reading cache

Checking package manager
Reading package lists... Done
Building dependency tree        
Reading state information... Done
Hit http://archive.ubuntu.com/ubuntu focal InRelease                           
Hit http://security.ubuntu.com/ubuntu focal-security InRelease                 
Hit http://archive.ubuntu.com/ubuntu focal-updates InRelease                   
Fetched 0 B in 0s (0 B/s)                                                      
Reading package lists... Done    
Building dependency tree          
Reading state information... Done
None

Checking for installed snaps

Calculating snap size requirements

Updating repository information
Hit http://security.ubuntu.com/ubuntu jammy-security InRelease                 
Hit http://archive.ubuntu.com/ubuntu jammy InRelease                           
Hit http://archive.ubuntu.com/ubuntu jammy-updates InRelease                   
Fetched 0 B in 0s (0 B/s)

Checking package manager
Reading package lists... Done    
Building dependency tree          
Reading state information... Done

Calculating the changes

Traceback (most recent call last):
  File "./jammy", line 8, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-wah0wx3w/DistUpgrade/DistUpgradeMain.py", line 241, in main
    if app.run():
  File "/tmp/ubuntu-release-upgrader-wah0wx3w/DistUpgrade/DistUpgradeController.py", line 2042, in run
    return self.fullUpgrade()
  File "/tmp/ubuntu-release-upgrader-wah0wx3w/DistUpgrade/DistUpgradeController.py", line 1939, in fullUpgrade
    if not self.askDistUpgrade():
  File "/tmp/ubuntu-release-upgrader-wah0wx3w/DistUpgrade/DistUpgradeController.py", line 1139, in askDistUpgrade
    res = self._view.confirmChanges(_("Do you want to start the upgrade?"),
  File "/tmp/ubuntu-release-upgrader-wah0wx3w/DistUpgrade/DistUpgradeViewText.py", line 212, in confirmChanges
    print(2/0)
ZeroDivisionError: division by zero

After this, apport prompted me to report a bug as expected.

tags:

added: verification-done verification-done-jammy
removed: verification-needed verification-needed-jammy

Revision history for this message

Launchpad Janitor (janitor) wrote on 2022-12-06:

This bug was fixed in the package ubuntu-release-upgrader - 1:22.04.15

---------------
ubuntu-release-upgrader (1:22.04.15) jammy; urgency=medium

  * DistUpgrade: Remove firefox from deb2snap.json (LP: #1964036)
  * DistUpgrade: Just pass filename to apport report (LP: #1985964)
  * Run pre-build.sh: updating mirrors and translations.

-- Nick Rosbrook <email address hidden> Fri, 14 Oct 2022 10:29:36 -0400

Changed in ubuntu-release-upgrader (Ubuntu Jammy):
status:	Fix Committed → Fix Released

Revision history for this message

Brian Murray (brian-murray) wrote on 2022-12-06: Update Released

#10

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntuubuntu-release-upgrader package

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Bug Description

Related branches

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
ubuntu-release-upgrader package