mysql server crashes during install/update: Error: Unable to shut down server with process id X

Hi,
from your logs:

Error: Unable to shut down server with process id 21763

2020-10-10T05:35:05.369830Z 0 [ERROR] [MY-011292] [Server] Plugin mysqlx reported: 'Preparation of I/O interfaces failed, X Protocol won't be accessible'
2020-10-10T05:35:05.370488Z 0 [ERROR] [MY-011300] [Server] Plugin mysqlx reported: 'Setup of socket: '/var/run/mysqld/mysqlx.sock' failed, can't create lock file /var/run/mysqld/mysqlx.sock.lock'

It reminds me of a different transient issue that occurred to some users in the past.
Is this issue reproducible or does it go away on a retry of the package upgrade?

P.S. the reason it is shown "every time" might be due to an upgrade tried "every time". In that case chances are that a local config change made it unable to start but a service restart is required for an upgrade. Was there any config change that could explain that?

Thank you for taking the time to report this bug and helping to make Ubuntu better.

On upgrading a service this service has to be restarted to pick up the fixes.
Rather rarely a real issue occurs that the newer version does e.g. fail with the formerly working configuration.
But most of the time what happens is, that a service was installed, but stays unconfigured or experimented with but left in a broken state.

Now on any update of the related packages that service has to be restarted, but since its config is incomplete/faulty it fails to restart.
Therefore the update of that package has to consider itself incomplete.

Depending on your particular case there are two solutions:
- either remove the offending package if you don't want to continue using it.
- Or if you do want to keep it please fix the configuration so that re-starting the service will work.

Since it seems likely to me that this is a local configuration problem, rather than a bug in Ubuntu, I'm marking this bug as Incomplete.

If indeed this is a local configuration problem, you can find pointers to get help for this sort of problem here: http://www.ubuntu.com/support/community

Or if you believe that this is really a bug, then you may find it helpful to read "How to report bugs effectively" http://www.chiark.greenend.org.uk/~sgtatham/bugs.html. We'd be grateful if you would then provide a more complete description of the problem, explain why you believe this is a bug in Ubuntu rather than a problem specific to your system, and then change the bug status back to New.

@Lars - I subscribed you as I've seen this particular kind of lock issue on upgrade a few times recently. I wanted to ask if this is any new/old issue that has (for which reason?) a comeback to be more common now?

iirc, we had a bug earlier where the apparmor profile was missing the mysqlx.sock file (new in 8.0), but that doesn't seem to be the case here, and no apparmor errors on it in the dmesg file.

I'm not sure if it's relevant, but there's an apparmor error for snap with docker and mysql mentioned. The error itself doesn't tell me much, but can you verify that you're not running mysql in a snap or docker container with socket files and/or ports mapped to the host? Another process already having written the mysql.sock file might explain the issue.

I'll do some testing. I'm not 100% sure the mysqlx.sock error is fatal. The server log seems to end with a successful startup.

From terminal:

mysqld is running as pid 21763
Error: Unable to shut down server with process id 21763
dpkg: error processing package mysql-server-8.0 (--configure):
 installed mysql-server-8.0 package post-installation script subprocess returned error exit status 1

Looking at that part of the postinst code, we have some oold logic for doing server shutdowns that waits a second for mysqld to stop, and 8.0 is slower to startup and shutdown than older versions, so it may simply be hitting a timeout without there strictly being anything wrong.

Typo: It waits 60 seconds, not a second

[Expired for mysql-8.0 (Ubuntu) because there has been no activity for 60 days.]

FYI - I've dup'ed a few more bugs that I've seen on triage onto this one.
This shall help to either better find the discussion that was had so far and to ensure all further comments on the same case happen here.

More duplicates sent this way.

I'm tagging this bug as server-todo because I believe it is actionable and it is also the root cause of several of our reports.

I believe the first step here would be to try to reproduce this but (I couldn't so far), and then coming up with a simple patch to extend the current timeout that's present on the stop_server routine on mysql-server-8.0.postinst file. This patch should likely be sent to the Debian mysql package so that Lars can take a look there.

This bug was fixed in the package mysql-8.0 - 8.0.28-0ubuntu3

---------------
mysql-8.0 (8.0.28-0ubuntu3) jammy; urgency=medium

  * d/a/source_mysql-8.0.py: Improve apport hook
    - Update call signature for add_info() (LP: #1922413)
    - Skip non-existing conf dirs (LP: #1958641)
    - If /etc/mysql missing, ask confirmation (LP: #1922412)
    - Fix all lint/flakes warnings
      + python3 -m py_compile source_mysql-8.0.py
      + pyflakes3 source_mysql-8.0.py
      + pylint source_mysql-8.0.py

 -- Bryce Harrington <email address hidden> Wed, 16 Feb 2022 19:07:41 -0800

This bug was fixed in the package mysql-8.0 - 8.0.28-0ubuntu4

---------------
mysql-8.0 (8.0.28-0ubuntu4) jammy; urgency=medium

  * d/mysql-server-8.0.postinst: Increase stop_server timeout so shutdowns that
    last up to 3 minutes do not trigger apport (LP: #1899248)

 -- Lena Voytek <email address hidden> Thu, 10 Mar 2022 11:57:56 -0700

Hello Shubham, or anyone else affected,

Accepted mysql-8.0 into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-8.0/8.0.28-0ubuntu0.21.10.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Shubham, or anyone else affected,

Accepted mysql-8.0 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-8.0/8.0.28-0ubuntu0.20.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Verified for focal on version 8.0.28-0ubuntu0.20.04.4 with the following test:

lxc launch images:ubuntu/focal test-mysql
lxc exec test-mysql bash

apt update && apt dist-upgrade -y

cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

apt update

apt install -y mysql-server

grep mysqlx /var/log/mysql/error.log

Verified for impish on version 8.0.28-0ubuntu0.21.10.4 with the following test:

lxc launch images:ubuntu/impish test-mysql
lxc exec test-mysql bash

apt update && apt dist-upgrade -y

cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

apt update

apt install -y mysql-server

grep mysqlx /var/log/mysql/error.log

The verification of the Stable Release Update for mysql-8.0 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package mysql-8.0 - 8.0.28-0ubuntu0.21.10.4

---------------
mysql-8.0 (8.0.28-0ubuntu0.21.10.4) impish; urgency=medium

  * d/mysql-server-8.0.postinst:
    - Increase stop_server timeout so shutdowns that last up to 3 minutes do not
      trigger apport (LP: #1899248)
    - Create and add correct permissions to the mysql socket directory before
      running mysqld post-update. (LP: #1964969)

 -- Lena Voytek <email address hidden> Mon, 14 Mar 2022 07:35:49 -0700

Someone should look at the focal arm64 FTBFS. It's been recently retried and failed with SIGKILL, suggesting OOM, so it probably needs more than just a no-thought hitting of the retry button.

This bug was fixed in the package mysql-8.0 - 8.0.29-0ubuntu0.20.04.2

---------------
mysql-8.0 (8.0.29-0ubuntu0.20.04.2) focal-security; urgency=medium

  * SECURITY UPDATE: Update to 8.0.29 to fix security issues
    - debian/control, debian/tests/control: add libexpect-perl as required
      by new test.
    - debian/mysql-testsuite-8.0.install, debian/mysql-router.install:
      added new files.
    - CVE-2022-21412, CVE-2022-21413, CVE-2022-21414, CVE-2022-21415,
      CVE-2022-21417, CVE-2022-21418, CVE-2022-21423, CVE-2022-21425,
      CVE-2022-21427, CVE-2022-21435, CVE-2022-21436, CVE-2022-21437,
      CVE-2022-21438, CVE-2022-21440, CVE-2022-21444, CVE-2022-21451,
      CVE-2022-21452, CVE-2022-21454, CVE-2022-21457, CVE-2022-21459,
      CVE-2022-21460, CVE-2022-21462, CVE-2022-21478
  * d/rules: increase build-time test timeouts to fix test failures that
    were resulting in FTBFS.
  * d/rules: Disable parallel builds on arm64

 -- Marc Deslauriers <email address hidden> Mon, 02 May 2022 07:07:07 -0400