libmysqlclient21 use of /usr/share/mysql collides with mariadb-server-core-10.3's use of the same directory

Bug #1877504 reported by Kim Tyler
308
This bug affects 8 people
Affects Status Importance Assigned to Milestone
mariadb-10.3 (Debian)
Fix Released
Unknown
mariadb-10.3 (Ubuntu)
Invalid
Undecided
Unassigned
mysql-8.0 (Ubuntu)
Triaged
Undecided
Unassigned

Bug Description

after upgrade libmysqlclient21 from 8.0.19-0ubuntu5 to 8.0.20-0ubuntu0.20.04.1, mythtv-setup, mythfrontend, mythbackend and mythpreviewgen all segfault when exiting.(and probably others) In particular, mythfrontend will restart after "crash" (segfault). Reverting libmysqlclient21 back to version 8.0.19-0ubuntu5 fixes the problem.

The latest version is a security fix, so presumably this is a security issue as we can't install it without myth segfaulting.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: libmysqlclient21 8.0.19-0ubuntu5
ProcVersionSignature: Ubuntu 5.4.0-29.33-generic 5.4.30
Uname: Linux 5.4.0-29-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu27
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
Date: Fri May 8 14:20:38 2020
InstallationDate: Installed on 2020-04-29 (9 days ago)
InstallationMedia: Ubuntu 20.04 LTS "Focal Fossa" - Release amd64 (20200423)
SourcePackage: mysql-8.0
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Kim Tyler (ktyler) wrote :
Revision history for this message
Leonidas S. Barbosa (leosilvab) wrote :

Hi Kim,

Thanks for report this issue. Could you please provides detailed steps in how to reproduce it?

information type: Private Security → Public Security
Revision history for this message
Kim Tyler (ktyler) wrote : Re: [Bug 1877504] Re: new version of libmysqlclient21 8.0.20-0ubuntu0.20.04.1 causes mythtv-set, mythbackend and mythfrontend to segfault on exit.

After the release of Focal, I did a fresh install with mariadb server.
I did a mysqldump backup of the existing mythtv (bionic) mariadb
database and installed mythtv (mythbackend + mythfrontend + plugins) and
configured it to work with the current setup.

This all worked almost perfectly, just needing minor tweaks.

I have been installing updates to the system mostly when they are available.

I noticed that after an update to the system just recently, the
mythfrontend was giving a segfault when exiting, causing the crash to be
detected and mythfrontend to restart.  Further investigation showed
indications that libmysqlclient was involved.

I downgraded to the previous version of libmysqlclient, and the problem
ceased. All was working as previously.

Upgrading to the latest (security fix) caused the problem to start
again.  I downgraded again and locked the version, and the problem went
away again.  I don't have any external facing mysql connections, so
security is not an imperative at the moment, but it does make me nervous
not to have security updates.

To cause the problem :- install the latest mysqlclient

To workaround the problem : downgrade the mysqlclient

If you need me to test anything, please let me know.

Good luck, and thanks,

Kim

On 9/5/20 1:18 am, Leonidas S. Barbosa wrote:
> Hi Kim,
>
> Thanks for report this issue. Could you please provides detailed steps
> in how to reproduce it?
>

Revision history for this message
Paul Harrison (mythtv-paulh) wrote : Re: new version of libmysqlclient21 8.0.20-0ubuntu0.20.04.1 causes mythtv-set, mythbackend and mythfrontend to segfault on exit.

This simple test application may be easier for you to reproduce this bug.

You will have to change the host,user name,password and database name to valid options for you but once you do that the test app will crash every time it exits.

Revision history for this message
Paul Harrison (mythtv-paulh) wrote :

Sample run under gdb with backtrace showing the crash with the test app.

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

Is this when using MariaDB or when using MySQL?

Revision history for this message
Paul Harrison (mythtv-paulh) wrote :

My tests were done with a remote server running mariadb-server-10.1 running on a default Ubuntu 18.04.4 LTS server.

The client side was running on Ubuntu 19.10 but others have also reported problems on 20.04 LTS.

As Kim has said downgrading to libmysqlclient21 back to 8.0.17 does fix the problem.

We have one user who has confirmed that the issue is not present when using mysql 8 instead of Mariadb.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in mysql-8.0 (Ubuntu):
status: New → Confirmed
Revision history for this message
Kim Tyler (ktyler) wrote :

I can confirm that the test program above causes segfault with current libmysqlclient21 but not with previous version, using mariadb as the server.(sorry, it took me a while to get all the dependencies together to compile and execute the program.)

Revision history for this message
Paul Harrison (mythtv-paulh) wrote :

Attached is a new reproducer that just uses the libmysqlclient directly without using Qt.

It's based on an example from a zetcode tutorial.

I used this to compile it :-
gcc mysql_test.c -o mysql_test -I/usr/include `mysql_config --cflags --libs`

The interesting thing is if you comment out both mysql_server_end() and mysql_library_end() at the bottom of the example it works without segfault. Uncomment either call and it will segfault.

The mysql docs say mysql_server_end() is depreciated and new code should use mysql_library_end() but they both cause a segfault. Ref: https://dev.mysql.com/doc/refman/8.0/en/mysql-server-end.html

This is the bt with debugging symbols installed for libmysqlclient which is a little more informative and should help a MySQL dev pinpoint the problem.

(gdb) bt
#0 0x00007ffff793823f in charset_uninit () at ./mysys/charset.cc:925
#1 0x00007ffff793d753 in my_end (infoflag=infoflag@entry=0) at ./mysys/my_init.cc:200
#2 0x00007ffff78db173 in mysql_server_end () at ./libmysql/libmysql.cc:202
#3 0x00005555555554f0 in main ()

I'll also attached a bt with bt full in case it's useful.

Revision history for this message
Paul Harrison (mythtv-paulh) wrote :

Attached a bt full of the crash

Revision history for this message
Lars Tangvald (lars-tangvald) wrote :

Thanks for the test case. Connecting to a mariadb server isn't supported (and thus not tested at all), but segfaulting is pretty bad, so hopefully we can get this fixed with all the information you've provided.

It's important to note, though, that MariaDB and MySQL diverge more and more with every release, and at some point they will most likely no longer work together at all.

Revision history for this message
Lars Tangvald (lars-tangvald) wrote :

Hm, if I install upstream's libmysqlclient21_8.0.20, there's no segfault.

Revision history for this message
Lars Tangvald (lars-tangvald) wrote :

This is probably the same as https://bugs.launchpad.net/ubuntu/+source/qtbase-opensource-src/+bug/1877183

I'll experiment a bit and see if maybe there are any compiler flags we use that are causing this.

Revision history for this message
Paul Harrison (mythtv-paulh) wrote :

I've done a little more debugging. Hopefully this output from a gdb session helps.

The segfault occurs here because in this case cs->coll is NULL
https://github.com/mysql/mysql-server/blob/8.0/mysys/charset.cc#L925

I don't know where the latin2_general_nopad_ci character set is coming from since I don't see any reference to it in the upstream MySQL sources on GitHub but I could be looking at the wrong sources.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff793823f in charset_uninit () at ./mysys/charset.cc:925
925 ./mysys/charset.cc: No such file or directory.
(gdb) bt
#0 0x00007ffff793823f in charset_uninit () at ./mysys/charset.cc:925
#1 0x00007ffff793d753 in my_end (infoflag=infoflag@entry=0) at ./mysys/my_init.cc:200
#2 0x00007ffff78db173 in mysql_server_end () at ./libmysql/libmysql.cc:202
#3 0x0000555555555513 in main ()
(gdb) print cs
$1 = (CHARSET_INFO *) 0x5555555a3e00
(gdb) print cs->coll
$2 = (MY_COLLATION_HANDLER *) 0x0
(gdb) print cs->csname
$3 = 0x5555555a3ef0 "latin2"
(gdb) print cs->name
$4 = 0x5555555a3ef8 "latin2_general_nopad_ci"
(gdb) print cs->comment
$5 = 0x5555555a3ed0 "ISO 8859-2 Central European"
(gdb) print cs->number
$6 = 1033
(gdb) print cs->state
$7 = 49
(gdb) print cs->primary_number
$8 = 0
(gdb) print cs->binary_number
$9 = 0
(gdb) print cs->ctype
$10 = (const uchar *) 0x0
(gdb) print cs->to_lower
$11 = (const uchar *) 0x0
(gdb) print cs->to_upper
$12 = (const uchar *) 0x0
(gdb) print cs->cset
$13 = (MY_CHARSET_HANDLER *) 0x0

Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

From:

bool init_compiled_charsets(myf flags MY_ATTRIBUTE((unused)))

there are charsets being added to "all_charsets" (global variable) through:

void add_compiled_collation(CHARSET_INFO *cs) {
  DBUG_ASSERT(cs->number < array_elements(all_charsets));
  all_charsets[cs->number] = cs;
  map_coll_name_to_number(cs->name, cs->number);
  map_cs_name_to_number(cs->csname, cs->number, cs->state);
  cs->state |= MY_CS_AVAILABLE;
}

and that contain NO COLLATION, this way.. when uniting them:

void charset_uninit() {
  for (CHARSET_INFO *cs : all_charsets) {
    if (cs && cs->coll->uninit) {
      cs->coll->uninit(cs);
    }
  }

cs->coll is NULL. This is an upstream bug.. would you like to report ? You can report upstream and link it here so we can backport a fix whenever someone fixes it. Maybe @lars can tackle it.

Revision history for this message
Paul Harrison (mythtv-paulh) wrote :

Sorry I don't have an Oracle login to report the bug upstream and don't want one. Hopefully someone else can report it to them.

Revision history for this message
Lars Tangvald (lars-tangvald) wrote :

Thanks for looking into it more. This might be what the devs need, so I'll pass it on (I've been trying to figure out exactly what causes it to work with upstream's package (or a libmysqlclient21 built just plainly with cmake, make, make install), but no luck.

Revision history for this message
Lars Tangvald (lars-tangvald) wrote :

Simply doing "if (cs && cs->coll && cs->coll->uninit) {" fixes the segfault in the test case, so I just need to check with a dev if that's a sane way to handle this (or if I'd just be hiding the symptom)

Paride Legovini (paride)
tags: added: update-excuse
tags: added: regression-update
Revision history for this message
Paride Legovini (paride) wrote :

This is preventing mysql-8.0 (8.0.20-0ubuntu1) from migration from groovy-proposed to groovy, so I added the update-excuse tag. I also added the regression-update tag as this bug landed in Focal with the 8.0.20-0ubuntu0.20.04.1 security upload of mysql-8.0.

Revision history for this message
Paride Legovini (paride) wrote :

The root cause of this problem seems to be the fact that libmysqlclient21 from mysql-8.0 tries to use charset files from mariadb, but they are not compatible with it and lead to a segfault. More details in this Debian bug I filed:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961672

Revision history for this message
Paride Legovini (paride) wrote :

As this seems to be a bug on the mariadb side rather than on the mysql side, I filed this MP to unblock the groovy mysql-8 migration:

https://code.launchpad.net/~legovini/britney/mysql-8/+merge/384660

Let's see if I've got all the bits right :)

Revision history for this message
Lars Tangvald (lars-tangvald) wrote :

Thanks for the help! I'm running build&test with the patch I have to prevent the segfault as well, so it shouldn't be quite as vulnerable to this.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in mariadb-10.3 (Ubuntu):
status: New → Confirmed
Revision history for this message
Otto Kekäläinen (otto) wrote :

I would suggest that you put proper debian/control conflicts for libmysqlclient21 to signify that things that worked with libmysqlclient20 might stop working. If libmysqlclient21 is incompatible with libmysqlclient20 ways of doing things, it will automatically be incompatible with any previous version of MySQL or MariaDB (which continue to follow the pre-existing conventions).

Revision history for this message
Otto Kekäläinen (otto) wrote :

Has anybody tested using libmariadb3? There are no backwards incompatible changes in libmariadb3 - MythTV should run just fine with libmariadb3.

If there will be backwards incompatible updates the library will be dumped to libmariadb3. If Oracle MySQL has now in libmysqlclient21 version 8.0.20 some backwards incompatible changes, maybe it should consider a version bump?

Revision history for this message
Lars Tangvald (lars-tangvald) wrote :

@Otto The usr/share/mysql/charsets/Index.xml file from mysql 5.7 (or older versions of 8.0) does not cause this issue. MySQL does not and has never had any MariaDB compatibility, and can not be expected to work properly if internal MySQL files are replaced by ones from MariaDB.

That said, this is still a regression bug in MySQL's charset parser (not a deliberate change), which we are fixing (afaik, the charset definition that triggers this is not invalid).

Revision history for this message
Paride Legovini (paride) wrote :

Thanks Lars. I dropped the britney hint MP.

Revision history for this message
Paride Legovini (paride) wrote :

After some more thinking I think it could be worth unblocking the mysql migration sooner than later. My rationale is explained here:

https://code.launchpad.net/~legovini/britney/mysql-8/+merge/384720

Weighting it is up to the release team.

Robie Basak (racb)
summary: - new version of libmysqlclient21 8.0.20-0ubuntu0.20.04.1 causes mythtv-
- set, mythbackend and mythfrontend to segfault on exit.
+ libmysqlclient21 use of /usr/share/mysql collides with mariadb-server-
+ core-10.3's use of the same directory
Revision history for this message
Otto Kekäläinen (otto) wrote :

MySQL 8.0 packaging in Ubuntu should put new *.sys files into /usr/share/mysql-8.0 to make it possible to have old and new libraries installed *side by side*.

To stress it, this is not a MariaDB issue. This is about MySQL introducing a backwards incompatible change and it needs to adjust for it so it is MySQL users can upgrade smoothly from old MySQL installs to new MySQL installs.

Changed in mariadb-10.3 (Ubuntu):
status: Confirmed → Invalid
Changed in mariadb-10.3 (Debian):
status: Unknown → New
Revision history for this message
Chris Shelton (cshelton1409) wrote :

With the recent upgrade of libmysqlclient21 to version 8.0.21-0ubuntu0.20.04.3, this issue seems to be fixed for me. There are no longer any segmentation faults at the completion of MythTV related processes since the installation of this update earlier today on my system running Ubuntu 20.04.

Revision history for this message
Paul Harrison (mythtv-paulh) wrote :

The crash affecting MythTV is fixed by this patch

 * debian/patches/charset_file_crash.patch: don't crash on malformed
    charset files in mysys/charset.cc (LP: #1884809)

from this bug
https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1884809

I think the other issues with Mysql and MariaDB colliding still remains though as hinted at in the bug description for #1884809?

Revision history for this message
Kim Tyler (ktyler) wrote :

Original crashing of mythtv now fixed on ubuntu 20.04 libmysqlclient21 version 8.0.21-0ubuntu0.20.04.3 with mariadb server.

Revision history for this message
Otto Kekäläinen (otto) wrote :

> I think the other issues with Mysql and MariaDB colliding still remains though as hinted at in the bug description for #1884809?

MySQL collides with it's own previous versions. The Ubuntu packaging of MySQL should start using a new path name like the MySQL.com packaging started doing, see https://jira.mariadb.org/browse/MDEV-22735?focusedCommentId=154508&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-154508

Revision history for this message
Robie Basak (racb) wrote :

Hi Otto,

> MySQL collides with it's own previous versions.

I don't understand how this the case. Please could you expand on this?

Changed in mariadb-10.3 (Debian):
status: New → Fix Released
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi,
coming by while looking at bugs dormant for a while.

@Robie / Lena - when reading the reference I think it outlines that conflict well.
As far as i understand it:

We would - for example - in Ubuntu have a problem with an old libmysqlclient that is installed along a new mysql-8 that brings newer incompatible charsets to /usr/share/mysql.

The suggestion is to use (since being incompatible) versioned paths like /usr/share/mysql-8.0 which is a path in which older client lib (or anything else not bein 8.0) would never look.

I assume that is in our d/rules -DINSTALL_MYSQLSHAREDIR=share/mysql which could as well be version specific avoiding the clash.
Maybe also worth a look while at it -DINSTALL_SUPPORTFILESDIR=share/mysql

The mentioned upstream defaults are in cmake/install_layout.cmake and all are like ...mysql-${MYSQL_BASE_VERSION}...

Keeping the bug open and - unless I misunderstood - setting it to triaged.
One might have a new and deeper look now that the comment brings the case back up for attention.

Changed in mysql-8.0 (Ubuntu):
status: Confirmed → Triaged
Revision history for this message
Robie Basak (racb) wrote :

> We would - for example - in Ubuntu have a problem with an old libmysqlclient that is installed along a new mysql-8 that brings newer incompatible charsets to /usr/share/mysql.

Isn't this just an assumption? AIUI, the only thing that is clear is that MariaDB doesn't understand the latest data schema for /usr/share/mysql that MySQL uses.

On "old libmysqlclient", it's completely normal for /usr/share/* to update schemas in a way that older versions might not understand. What matters is that we (neither Debian nor Ubuntu) do not ship a package in a single distribution release that reads but doesn't understand it, including during an upgrade.

AIUI, this is just MariaDB? If MariaDB can't understand MySQL's /usr/share/mysql/ schema, surely it should use a different directory?

tags: removed: update-excuse
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.