mysql timeoutsec results in killing mysql process

Bug #1882527 reported by Warwick Bruce Chapman
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
mysql-5.7 (Ubuntu)
Fix Released
Undecided
Paride Legovini
Bionic
Fix Released
Undecided
Paride Legovini
mysql-8.0 (Ubuntu)
Fix Released
High
Unassigned
Eoan
Won't Fix
Medium
Unassigned
Focal
Fix Released
Medium
Paride Legovini

Bug Description

[Impact]

mysql-server-5.7.mysql.service (bionic) and mysql-server-8.0.mysql.service (focal) have a TimeoutSec=600. This has the effect of killing the MySQL process if this timeout is reached.

Very large databases can exceed the 600s timeout, and a safe tradeoff between timing out at some point and waiting long enough to accommodate large/huge databases does not seem to exist.

This issue has been fixed in Debian and in Ubuntu >=Hirsute by disabling the timeout (TimeoutSec=infinity).

[Test Plan]

This is probably the most interesting bit of the SRU :-)

In order to test this for real, as opposed to testing systemd's TimeoutSec, we need to make mysql very slow when it loads its tables. One way it to actually have huge tables, but I have no idea on how big they'd need to be. The other way is to slow down access to /var/lib/mysql at the I/O level. Something on these lines:

  apt install mysql-server-8.0
  systemctl stop mysql
  cd /var/lib
  mv mysql mysql.bak
  truncate -s 300M mysql.blk
  losetup --show --find mysql.blk

  dmsetup create slowdev --table \
    "0 $(blockdev --getsz /dev/loopX) delay /dev/loopX 0 100"
  # With /dev/loopX as printed by losetup, 100 = 100ms r/w delay
  # See: https://www.kernel.org/doc/Documentation/device-mapper/delay.txt

  mkfs.ext4 /dev/mapper/slowdev
  mkdir mysql
  mount /dev/mapper/slowdev mysql
  chown mysql:mysql mysql
  cp -av mysql.bak/* mysql

  time systemctl start mysql # slow!
  systemctl status mysql

By tuning the delay parameter it is possible to trigger the timeout with the pre-SRU package, and verify that post-SRU it can load in more than 10 minutes.

Note: this can be tested in an LXD VM but it requires booting the linux-image-generic kernel, as linux-image-kvm doesn't ship the dm delay target.
The lxd-agent won't work, just ssh-import-id and ssh in.

(I think this is overkill for this SRU, but it's a quite general way to make stuff slow. I'm sure it will come useful in other cases!)

[Where problems could occur]

The TimeoutSec=infinity syntax is supported by the systemd versions in all the supported releases of Ubuntu, so this won't be a problem.

Then only change in behavior due to this change will happen on systems where the timeout is reached, and mysql is thus killed. In these cases the database server wouldn't be running in any case, but there could be cases of bad or overgrown databases (e.g. because of a runaway script adding infinite data) where the timeout is doing the right thing, preventing mysql from consuming system resources forever. In these already broken systems TimeoutSec=infinity may increase the breakage. This won't affect working production systems.

[Development Fix]
[Stable Fix]

The same fix is already already landed in Hirsute, Impish and Debian unstable.

[Original Description]

MySQL on 20.04 has TimeoutSec set to 600 (IIRC) in the systemd script. This has the effect of killing the MySQL process if this timeout is reached.

IMHO this is a Very Bad Idea. A database server process should only be force killed by a user action.

I would prefer that the server had unlimited time to cleanly shutdown and startup (eg if recovering).

Our DB is about 500GB with some very large tables (for us at least) eg. 250GB and we've had more than a few unfortunate delays as a result of delayed startup caused by recoveries because MySQL was killed prematurely.

Because MySQL 8.0 has reduced the default logging level, it was not clear to me that the process was being force killed.

I believe the MySQL team are of the same view as me per https://bugs.mysql.com/bug.php?id=91423:

```
[12 Jul 2019 15:57] Paul Dubois
Posted by developer:

Fixed in 8.0.18.

On Debian, long InnoDB recovery times at startup could cause systemd
service startup failure. The default systemd service timeout is now
disabled (consistent with RHEL) to prevent this from happening.
```

Related branches

Revision history for this message
Warwick Bruce Chapman (warwickchapman) wrote :

I see I have created this bug report in xorg instead of MySQL. Please accept my apology and either move or delete. I am happy to recreate, just tell me which.

affects: xorg-server (Ubuntu) → mysql-8.0 (Ubuntu)
tags: added: focal
Revision history for this message
Warwick Bruce Chapman (warwickchapman) wrote :

Thanks Daniel.

Revision history for this message
Balint Harmath (bharmath) wrote :

The vendor is working on it - we can assist to that.

Changed in mysql-8.0 (Ubuntu):
status: New → Confirmed
no longer affects: mysql-8.0 (Ubuntu Groovy)
Changed in mysql-8.0 (Ubuntu Focal):
status: New → Triaged
Changed in mysql-8.0 (Ubuntu Eoan):
status: New → Triaged
Changed in mysql-8.0 (Ubuntu Bionic):
status: New → Triaged
importance: Undecided → Medium
Changed in mysql-8.0 (Ubuntu Eoan):
importance: Undecided → Medium
Changed in mysql-8.0 (Ubuntu Focal):
importance: Undecided → Medium
Changed in mysql-8.0 (Ubuntu):
importance: Undecided → High
tags: added: server-next
Robie Basak (racb)
tags: added: bitesize
Revision history for this message
Brian Murray (brian-murray) wrote :

The Eoan Ermine has reached end of life, so this bug will not be fixed for that release

Changed in mysql-8.0 (Ubuntu Eoan):
status: Triaged → Won't Fix
Revision history for this message
Koen (koen-beek) wrote :

The mysql bug ticket (https://bugs.mysql.com/bug.php?id=91423) mentions

[12 Jul 2019 15:57] Paul Dubois

Posted by developer:

Fixed in 8.0.18.

On Debian, long InnoDB recovery times at startup could cause systemd
service startup failure. The default systemd service timeout is now
disabled (consistent with RHEL) to prevent this from happening.

--

the latest ubuntu code (based on 8.0.22) contains following mentions of TimeoutSec

koen@hpubuntu:~/Dev/personal/ubuntu/mysql$ grep -rR TimeoutSec *
mysql-8.0-8.0.22/packaging/deb-in/mysql-packagesource-server.mysql.service.in:TimeoutSec=0
mysql-8.0-8.0.22/packaging/deb-in/mysql-packagesource-server.mysql@.service.in:TimeoutSec=0
mysql-8.0-8.0.22/scripts/systemd/mysqld@.service.in:TimeoutSec=0
mysql-8.0-8.0.22/scripts/systemd/mysqld.service.in:TimeoutSec=0
mysql-8.0-8.0.22/debian/mysql-server-8.0.mysql.service:TimeoutSec=600

--> it seems there's still 1 mention of 600 seconds in the Debian/ubuntu package

I can take on this issue but it might take a few weeks forme to complete - ETA mid Jan 2021 as I'm pretty new at ubuntu bugfixing/packaging and I'll have some questions for some more expert people

- correct patch description
- how to involve Debian upstream
- how to make patches for several releases (I'll start with hirsute)

if it's more urgent don't hesitate to assign someone else

Changed in mysql-8.0 (Ubuntu):
assignee: nobody → Koen (koen-beek)
Revision history for this message
Koen (koen-beek) wrote :

There does not seem to be an equivalent debian bug about this issue yet

will make a patch in ubuntu and forward it to debian : https://wiki.ubuntu.com/Debian/Bugs

Revision history for this message
Koen (koen-beek) wrote :

hmm when doing a

git clone https://salsa.debian.org/mariadb-team/mysql.git -b mysql-8.0/debian/master

it seems it has been fixed by putting infinity to TimeoutSec (I suppose that's the same as 0) in Debian

mysql/debian/mysql-server-8.0.mysql.service:TimeoutSec=infinity

-> seems to have been fixed in Debian in mysql-8.0.0.22

and https://metadata.ftp-master.debian.org/changelogs//main/m/mysql-8.0/mysql-8.0_8.0.22-1_changelog

shows :

mysql-8.0 (8.0.22-1) unstable; urgency=medium
...
  * d/systemd: Disable service timeout
    For large databases, the service could timeout on stop, possibly
    leading to data corruption during a system shutdown.
    (LP: #1882527)

--> it's in debian in the next patch release

Ubuntu is currently on 8.0.22-0(ubuntu0.20.10.2) -> ubuntu just needs to align with debian 8.0.22-1 instead of 8.0.22-0

Revision history for this message
Koen (koen-beek) wrote :

I'm unassigning myself as I'm not sure what the next steps are but I believe it should be corrected by ubuntu taking the latest debian version of the package (i.e. mysql-8.0 (8.0.22-1)) -> no ubuntu-patch needed

Changed in mysql-8.0 (Ubuntu):
assignee: Koen (koen-beek) → nobody
Koen (koen-beek)
description: updated
Revision history for this message
Robie Basak (racb) wrote :

This is fixed in Hirsute now. Backports to Focal and Bionic may be appropriate.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mysql-8.0 - 8.0.23-1ubuntu2

---------------
mysql-8.0 (8.0.23-1ubuntu2) hirsute; urgency=medium

  * Add missing libzstd-dev on runtime dev package
    (Closes: #981905)

 -- Gianfranco Costamagna <email address hidden> Thu, 04 Feb 2021 22:30:12 +0100

Changed in mysql-8.0 (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Utkarsh Gupta (utkarsh) wrote :

> This is fixed in Hirsute now. Backports to Focal and Bionic may
> be appropriate.

Agreed. I'll ask Miriam or Paride to TAL. Thanks!

Paride Legovini (paride)
Changed in mysql-8.0 (Ubuntu Bionic):
assignee: nobody → Paride Legovini (paride)
Changed in mysql-8.0 (Ubuntu Focal):
assignee: nobody → Paride Legovini (paride)
Paride Legovini (paride)
description: updated
Paride Legovini (paride)
description: updated
Paride Legovini (paride)
description: updated
Paride Legovini (paride)
description: updated
no longer affects: mysql-5.7 (Ubuntu Eoan)
no longer affects: mysql-5.7 (Ubuntu Focal)
no longer affects: mysql-5.7 (Ubuntu Bionic)
Changed in mysql-5.7 (Ubuntu):
status: New → Triaged
no longer affects: mysql-8.0 (Ubuntu Bionic)
Changed in mysql-5.7 (Ubuntu):
assignee: nobody → Paride Legovini (paride)
Paride Legovini (paride)
description: updated
Revision history for this message
Robie Basak (racb) wrote :

Focal is frozen for 20.04.3 right now, so I'll accept this into Bionic only into proposed for now. However we should release Focal before Bionic to avoid regressing Bionic users upgrading to Focal.

Changed in mysql-5.7 (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello Warwick, or anyone else affected,

Accepted mysql-5.7 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.7/5.7.35-0ubuntu0.18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (mysql-5.7/5.7.35-0ubuntu0.18.04.2)

All autopkgtests for the newly accepted mysql-5.7 (5.7.35-0ubuntu0.18.04.2) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

postfix/3.3.0-1ubuntu0.3 (amd64, i386)
slurm-llnl/17.11.2-1build1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#mysql-5.7

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Paride Legovini (paride)
Changed in mysql-5.7 (Ubuntu Bionic):
assignee: nobody → Paride Legovini (paride)
Revision history for this message
Paride Legovini (paride) wrote :

I did the verification for mysql-5.7/bionic according to the [Test Plan], will proceed with the mysql-8.0/focal verification once it's accepted in -proposed.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Warwick, or anyone else affected,

Accepted mysql-8.0 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-8.0/8.0.26-0ubuntu0.20.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in mysql-8.0 (Ubuntu Focal):
status: Triaged → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Paride Legovini (paride) wrote :

Verification done for mysql-8.0/focal again following the [Test Plan], so I tagged this overall verification-done.

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (mysql-8.0/8.0.26-0ubuntu0.20.04.3)

All autopkgtests for the newly accepted mysql-8.0 (8.0.26-0ubuntu0.20.04.3) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

ruby-mysql2/0.5.2-1ubuntu3 (s390x, arm64, ppc64el, armhf, amd64)
kopanocore/8.7.0-7ubuntu1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#mysql-8.0

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Paride Legovini (paride) wrote (last edit ):

The reported Bionic regressions were false positives and have now been cleared. The Focal postfix regression was also a false positive, and it's not cleared.

One regression remains on Focal:

  ruby-mysql2/0.5.2-1ubuntu3

which isn't really triggered by the mysql-8.0 as it's reproducible with what we already have in the archive, e.g. by doing:

autopkgtest --no-built-binaries \
  --setup-commands="sed -i '/^# deb-src/s/^# //' /etc/apt/sources.list" \
  --setup-commands="apt-get update" \
  ruby-mysql2 -- lxd ubuntu:focal

Revision history for this message
Paride Legovini (paride) wrote :

The Bionic upload now has no issues preventing migration.

The Focal upload is being held back by: https://bugs.launchpad.net/ubuntu/+source/ruby-mysql2/+bug/1942476

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for mysql-5.7 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mysql-5.7 - 5.7.35-0ubuntu0.18.04.2

---------------
mysql-5.7 (5.7.35-0ubuntu0.18.04.2) bionic; urgency=medium

  * d/systemd: Disable service timeout
    For large databases, the service could timeout on stop, possibly
    leading to data corruption during a system shutdown.
    (LP: #1882527)

 -- Paride Legovini <email address hidden> Thu, 19 Aug 2021 18:34:52 +0200

Changed in mysql-5.7 (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mysql-8.0 - 8.0.26-0ubuntu0.20.04.3

---------------
mysql-8.0 (8.0.26-0ubuntu0.20.04.3) focal; urgency=medium

  * d/systemd: Disable service timeout
    For large databases, the service could timeout on stop, possibly
    leading to data corruption during a system shutdown.
    (LP: #1882527)

 -- Paride Legovini <email address hidden> Thu, 19 Aug 2021 15:32:33 +0200

Changed in mysql-8.0 (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Paride Legovini (paride) wrote :

mysql-5.7 is fixed anywhere we care; I'm marking the main task as Fix Released.

Changed in mysql-5.7 (Ubuntu):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.