mysql-server-5.7 postinst fails when in read-only mode

The mysql-server packages (8.0) provided in later releases (focal+) are not affected.

Hello Simon and thanks for your bug report and for the detailed steps to reproduce it. The command causing the error is mysql_upgrade, which can also be run manually, without triggering the execution of the postinst.

The command has been dropped in mysql-8, as the upgrade is now done automatically by the mysql daemon on startup.

What you are hitting seems to be a limitation of mysql/mysql_upgrade. We could avoid running mysql_upgrade if mysql is configured in (super-)read-only mode, but we'd need a reliable way to check if that's the case even if the daemon is not running, and still I'd not be really aware of the implications of skipping the mysql_upgrade runs.

While I acknowledge the situation is not ideal, I'm not sure on how we can safely improve on your setup.

Hi Paride, thanks for having a look. Would you consider the provided patch for inclusion? It worked in my local tests.

The attachment "Skip mysql_upgrade when in (super) read-only mode" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Hi Simon,

What's the full HA setup you're using here please? Would it be more appropriate for you to be using mysql-server-core-5.7 directly instead, with a manual configuration?

I don't think it's realistic for mysql-server apt/dpkg packaging to provide both automatic management of a database in /var/lib/mysql and also automatically support HA setups since apt/dpkg can only coordinate a single host at once. For that we have mysql-server-core-* packages with the expectation that users will configure what they need - either manually or with a higher level tool that can coordinate across multiple hosts.

While I have no objection to fixing specific cases, I wonder if it's worth the maintenance cost of including this case if it's to support a use case that we cannot realistically support anyway.

Your thoughts appreciated.

Hi Robie,

I took a look at mysql-server-core-5.7 based on IRC discussions with you. Unfortunately, this package doesn't create the mysql user and group, doesn't provide the systemd unit and doesn't integrate well with other tooling (like the mysql puppet module maintained by puppetlabs). For those reasons, I concluded it was better to stick with the non-core package for my client's setup.

I tried to lower the maintenance cost by providing a patch, but I would understand Canonical not wanting to adopt it for a user population that's unknown and potentially rather low.

Regards,
Simon

Hi Simon,

Would it help for you to manually trigger "freeze mode"? This was added as a way to stop automatic management of the service in case of some unrecoverable issue with it, without triggering apt errors.

If you create the file /etc/mysql/FROZEN (if you're installing in interactive mode you should get a debconf message about this), it should skip most of the postinst script, so it won't start the service, run mysql_upgrade, etc, but you should get all the unit and config files so you can easily init and start it up yourself.

Hi Lars,

The freeze mode looked promising but it bypasses a lot of good bits of the configure script like Apparmor profile activation, directories perms/owner fixes, etc. Also, the postinst script stops the MySQL service unconditionally and the configure aborts early in freeze mode, leaving the service stopped after the upgrade. Thanks for the suggestion though, much appreciated!

Regards,
Simon

Thank you for the discussion.

Since 8.0 isn't affected, I'm marking this as Fix Released with tasks for Xenial and Bionic only.

I'm on the fence about the suitability for your patch - as the packaging isn't intended to support HA configurations, so maintaining the patch incurs unnecessary maintenance burden.

However, as 8.0 isn't affected, the maintenance would only be necessary for Xenial and Bionic so would be minimal in this particular case, and your patch is non-invasive and relatively trivial.

If you're prepared to take that maintenance on, I'll support you landing SRUs for this (Bionic only if that's all you want, or Bionic and Xenial if you require Xenial). I'd expect you to follow the full SRU procedure please, apart for any necessary review and sponsorship which my team will take on for you.

Invalid because on second thought, src:mysql-5.7 doesn't actually exist in Focal or Groovy.

I should add that another reason I'm supportive of an SRU in principle is that working around is quite obtuse (it would need a switch from mysql-server-5.7 to mysql-server-core-5.7 only which is not trivial).

Hi Robie,

I filled the SRU template and would appreciate if you/your team could take a look at the debdiff for Bionic (I don't intent to get one for Xenial). I tested it, in various scenarios (R/W, R/O, super R/O) and it worked fine.

Regards,
Simon

Hi Simon,

Sorry, I must have missed your comment. A colleague pointed this bug out to me last week. Do you still need this? Would it still be useful to you, and would you still be able to drive SRU verification, etc? If so let me know and I'll review.

@Robie, yes I'm still interested in having this SRU'ed. Thanks for
getting back!

Reviewed and sponsored (rebasing on to 5.7.33-0ubuntu0.18.04.2). Now awaiting SRU review. Thanks!

It's quite common to get a flurry of maintainer script failure reports from users with broken MySQL installations after an update is released. I wonder if, given the timescale of this fix, we might prefer to stage this SRU after verification, until the next security update? Opinions appreciated.

Thanks you. Batching this with the next security update seems like the
best approach to me.

OK. Thanks!

Too bad 5.7.34-0ubuntu0.18.04.1 was released without the patch :/
I guess it will go in the next batch.

We'll need to rebase and reupload to the SRU queue. Sorry! Since I sponsored, I'm not supposed to do the SRU review myself so was waiting for another team member to do it, and that didn't happen before the security update arrived.

No problem, thanks Robie!

I went ahead and rebased and reuploaded it to the SRU queue. I'll also be reviewing and accepting it given that I've reuploaded the same changes Robie sponsored so my review of it really was as an SRU team member.

Hello Simon, or anyone else affected,

Accepted mysql-5.7 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.7/5.7.34-0ubuntu0.18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Thanks Brian, here are the verification steps (spoiler: it worked!) I followed from the SRU template:

$ lxc launch images:ubuntu/bionic sql1
Creating sql1
Starting sql1
$ lxc shell sql1

root@sql1:~# apt-get update && apt-get install -y --no-install-recommends mysql-server
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
...
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [371 kB]
Fetched 8,155 kB in 4s (2,201 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libaio1 libevent-core-2.1-6 libgdbm-compat4 libgdbm5 libnuma1 libperl5.26 libwrap0 mysql-client-5.7 mysql-client-core-5.7 mysql-common mysql-server-5.7 mysql-server-core-5.7 perl perl-modules-5.26 psmisc
Suggested packages:
  gdbm-l10n mailx tinyca perl-doc libterm-readline-gnu-perl | libterm-readline-perl-perl make
Recommended packages:
  libhtml-template-perl
The following NEW packages will be installed:
  libaio1 libevent-core-2.1-6 libgdbm-compat4 libgdbm5 libnuma1 libperl5.26 libwrap0 mysql-client-5.7 mysql-client-core-5.7 mysql-common mysql-server mysql-server-5.7 mysql-server-core-5.7 perl perl-modules-5.26 psmisc
0 upgraded, 16 newly installed, 0 to remove and 0 not upgraded.
Need to get 25.6 MB of archives.
After this operation, 195 MB of additional disk space will be used.
...
Setting up mysql-server-5.7 (5.7.34-0ubuntu0.18.04.1) ...
update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Renaming removed key_buffer and myisam-recover options (if present)
Created symlink /etc/systemd/system/multi-user.target.wants/mysql.service → /lib/systemd/system/mysql.service.
Setting up mysql-server (5.7.34-0ubuntu0.18.04.1) ...
Processing triggers for systemd (237-3ubuntu10.48) ...
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...

root@sql1:~# cat << EOF >> /etc/mysql/my.cnf
>
> [mysqld]
> super_read_only = ON
> read_only = ON
> EOF
root@sql1:~# service mysql restart
root@sql1:~# mysql -e "SELECT @@global.read_only, @@global.super_read_only;"
+--------------------+--------------------------+
| @@global.read_only | @@global.super_read_only |
+--------------------+--------------------------+
| 1 | 1 |
+--------------------+--------------------------+

# enable bionic-proposed
root@sql1:~# cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
> # Enable Ubuntu proposed archive
> deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
> EOF

root@sql1:~# apt-get update && apt-get dist-upgrade -V
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,754 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-proposed InRelease [242 kB]
Get:6 ht...

This bug was fixed in the package mysql-5.7 - 5.7.35-0ubuntu0.18.04.1

---------------
mysql-5.7 (5.7.35-0ubuntu0.18.04.1) bionic-security; urgency=medium

  * SECURITY UPDATE: Update to 5.7.34 to fix security issues
    - CVE-2021-2342, CVE-2021-2372, CVE-2021-2385, CVE-2021-2389,
      CVE-2021-2390

 -- Marc Deslauriers <email address hidden> Thu, 22 Jul 2021 08:02:24 -0400