release upgrade does not move to the new php apache mod

Note: I'll leave the two containers as-is for now in case you want any logs in them to be checked.

Hmm. Yes, seems pretty straightforward to reproduce the problem in
lxc. At the end of the upgrade it offers to remove mod php 7.2:

Reading state information... Done

Remove obsolete packages?

69 packages are going to be removed.

 Continue [yN] Details [d]d

Remove: libapache2-mod-php7.2

Remove (was auto installed) acl acpid btrfs-tools
  command-not-found-data dns-root-data dnsmasq-base ebtables
  gcc-8-base geoip-database libbind9-160 libdns-export1100 libdns1100
  libevent-2.1-6 libffi6 libfreetype6 libgdbm5 libgeoip1 libhogweed4
  libicu60 libidn11 libip4tc0 libip6tc0 libiptc0 libirs160
  libisc-export169 libisc169 libisccc160 libisccfg160 libjson-c3
  liblvm2app2.2 liblvm2cmd2.02 liblwres160 liblxc-common liblxc1
  libncurses5 libncursesw5 libnettle6 libnih1 libntfs-3g88
  libperl5.26 libplymouth4 libprocps6 libpython3.6
  libpython3.6-minimal libpython3.6-stdlib libreadline7 libssl1.0.0
  libtinfo5 lxcfs lxd lxd-client mlocate multiarch-support net-tools
  nplan perl-modules-5.26 php7.2-cli php7.2-common php7.2-json
  php7.2-opcache php7.2-readline python3-asn1crypto python3-pam
  python3.6 python3.6-minimal uidmap ureadahead xdelta3

I wonder if there needs to be a dependency added for the
libapache2-mod-php virtual package.

On Tue, Aug 04, 2020 at 09:47:14AM -0000, Christian Ehrhardt  wrote:
> Note: I'll leave the two containers as-is for now in case you want any
> logs in them to be checked.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1890263
>
> Title:
> release upgrade does not move to the new php apache mod
>
> Status in php7.4 package in Ubuntu:
> New
>
> Bug description:
> While checking how things behave for bug 1890029 I might have found
> another aspect/version of bug 1865218. But I'm not sure yet.
>
> Bryce was working on that and he probably has a better answer, so I'll
> file a bug about what I've seen and assign it to him to share his
> thoughts.
>
> I tried to ways of a bionic->focal upgrade in otherwise clear bionic
> LXD containers.
>
> I did:
> 1. get a bionic container
> 2. apt install libapache2-mod-php7.2
> that will pull in apache as well and work
>
> 3a) edited /etc/apt/sources.list from bionic -> focal
> apt update
> apt dist-upgrade
>
> This one left me with libapache2-mod-php7.2 of Bionic still installed
> and no 7.4 of Focal to be seen.
>
> 3b) I tried `do-release-upgrade -d` as some logic might be in there
>
> This was different, it removed libapache2-mod-php7.2 but didn't bring in 7.4 as I'd have expected.
> Before:
> root@b-to-f2:~# dpkg -l | grep -e php -e apache2
> ii apache2 2.4.29-1ubuntu4.13 amd64 Apache HTTP Server
> ii apache2-bin 2.4.29-1ubuntu4.13 amd64 Apache HTTP Server (modules and other binary files)
> ii apache2-data 2.4.29-1ubuntu4.13 all Apache HTTP Server (common files)
> ii apache2-utils 2.4.29-1ubuntu4.13 amd64 Apache HTTP Server (utility programs ...

There seem to be two different ways of installing modphp:

  1. sudo apt-get install libapache2-mod-php
or
  2. sudo apt-get install libapache2-mod-php7.2

libapache2-mod-php is a virtual package that depends on the appropriate libapache2-mod-phpN.M for that Ubuntu release; presumably if one had installed libapache2-mod-php the upgrade should work seamlessly. However, if one rather installed libapache2-mod-php7.2, then I think this is where the bug presents itself.

Furthermore, there's multiple upgrade methods to consider:

  A. Running `do-release-upgrade -d`
  B. Editing /etc/apt/sources.list from bionic -> focal (or focal -> jammy, or impish -> jammy)
  C. Upgrading just modphp itself via a PPA

Method C is presumably how most users would be experiencing this, so the main one needing fixed, but it's also the most time-intensive to test. #2 is easier to test and still conceivably used by some experienced users, so is worth getting right. #3 is easiest to test but pretty artificial (though php users are accustomed to upgrading php via ondrej's ppa).

We see that in each of these cases, libapache2-mod-php7.2 might be left in place, or removed as cleanup.

For testing method C, I've backported all the relevant PHP pieces to this PPA:

    https://launchpad.net/~bryce/+archive/ubuntu/modphp-upgrade-lp1890263-unfixed-backports

(C.1) Considering only bionic, installation style 1 works as expected:

$ sudo apt-get install libapache2-mod-php
The following NEW packages will be installed:
  apache2 apache2-bin apache2-data apache2-utils libapache2-mod-php libapache2-mod-php7.2 libaprutil1-dbd-sqlite3
  libaprutil1-ldap liblua5.2-0 libsodium23 php-common php7.2-cli php7.2-common php7.2-json php7.2-opcache
  php7.2-readline ssl-cert
0 upgraded, 17 newly installed, 0 to remove and 0 not upgraded.

$ sudo add-apt-repository -yus ppa:bryce/modphp-upgrade-lp1890263-unfixed-backports
...

$ sudo apt-get full-upgrade
The following packages were automatically installed and are no longer required:
  php7.2-cli php7.2-common php7.2-json php7.2-opcache php7.2-readline
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  libapache2-mod-php7.2
The following NEW packages will be installed:
  libapache2-mod-php7.4 libpcre2-8-0 php7.4-cli php7.4-common php7.4-json php7.4-opcache php7.4-readline
The following packages will be upgraded:
  libapache2-mod-php php-common

Notably, this successfully moved from the old modphp to the new one, but did not uninstall other php7.2 binary packages. The system ended with both php7.2 and php7.4, but only with libapache2-mod-php7.4. I think this is what we want.

To verify, I also purged all of the aforementioned packages, and re-installed libapache2-mod-php with the backport PPA still enabled. As expected and as desired, it directly installs libapache2-mod-php7.4 (7.4.3-4ubuntu2.11~bionic1).

(C.2) Next, in a clean new bionic container fully up to date, I installed the versioned modphp:

$ sudo apt-get install libapache2-mod-php7.2
The following NEW packages will be installed:
  apache2 apache2-bin apache2-data apache2-utils libapache2-mod-php7.2 libaprutil1-dbd-sqlite3 libaprutil1-ldap
  liblua5.2-0 libsodium23 php-common php7.2-cli php7.2-common php7.2-json php7.2-opcache php7.2-readline ssl-cert
0 upgraded, 16 newly installed, 0 to remove and 0 not upgraded.

Notice that one fewer package is installed in this situation, namely the virtual package libapache2-mod-php. (This fact will cause problems later.)

$ sudo add-apt-repository -yus ppa:bryce/modphp-upgrade-lp1890263-unfixed-backports
...

$ sudo apt-get full-upgrade
The following packages will be upgraded:
  php-common
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

So in upgrading with the PPA enabled, php-common gets upgraded but this does not trigger any php components to upgrade, so we're left with php7.2 and the old modphp.

Here we should note that some pieces come from phpM.N, others from php-common:

$ pkg-source php-common
php-defaults

$ pkg-source libapache2-mod-php
php-defaults

$ pkg-source libapache2-mod-php7.2
php7.2

$ pkg-source libapache2-mod-php7.4
php7.4

$ pkg-source libapache2-mod-php8.0
php8.0

$ pkg-source libapache2-mod-php8.1
php8.1

So, the fix for this issue will require updates to php-defaults on each release, and updates to the phpM.N package specific to each release.

Earlier I had speculated:

> I wonder if there needs to be a dependency added for the
> libapache2-mod-php virtual package.

However I don't think this will work as a solution, because already we have libapache2-mod-php depending on libapache2-mod-phpM.N, so adding a dependency in the opposite direction as well, would cause circular dependence.

This bug was fixed in the package php8.1 - 8.1.2-1ubuntu2

---------------
php8.1 (8.1.2-1ubuntu2) jammy; urgency=medium

  * d/control: Add transitional packages and Breaks/Replaces to force
    upgrade from earlier mod-php's to version 8.1.
    (LP: #1890263)
  * d/rules: Don't fill up build log with pedantic warnings.

 -- Bryce Harrington <email address hidden> Thu, 07 Apr 2022 17:46:26 +0000

I verified that the fix uploaded for jammy that's now available in the release, does indeed resolve the issue for impish:

###########################################
### Impish Test Case for BUGGY Behavior ###
###########################################
$ lxc launch ubuntu-daily:impish lp1890263-modphp-buggy-impish
$ lxc shell lp1890263-modphp-buggy-impish
# apt-get update; apt-get -y full-upgrade; apt-get -y autoremove
# reboot
$ lxc shell lp1890263-modphp-buggy-impish
# apt install -y libapache2-mod-php8.0
  The following NEW packages will be installed:
    apache2 ... libapache2-mod-php8.0 ... php8.0-common ...
  ...
# do-release-upgrade
  - Answer defaults for everything
  - Allow it to reboot when done, and log back in

# lsb_release -sc
jammy

# apt-cache policy libapache2-mod-php* | grep -B1 Installed:
libapache2-mod-php:
  Installed: 2:8.1+92ubuntu1
--
libapache2-mod-php7.4:
  Installed: (none)
--
libapache2-mod-php8.0:
  Installed: 8.1.2-1ubuntu2
--
libapache2-mod-php8.1:
  Installed: 8.1.2-1ubuntu2
--
libapache2-mod-php5:
  Installed: (none)

# result=$(apt-cache policy libapache2-mod-php8.1 | grep Installed: | cut -d: -f2 | xargs)
# [ "${result}" = "(none)" ] && echo "BUG DETECTED"

# logout

$ lxc stop lp1890263-modphp-buggy-impish
$ lxc delete lp1890263-modphp-buggy-impish

Due to this, we can close out the impish tasks for this bug, as solved.

Similarly, the focal -> jammy upgrade process is also confirmed as fixed now, with the jammy update:

###########################################
### Focal Test Case for BUGGY Behavior ###
###########################################
$ lxc launch ubuntu-daily:focal lp1890263-modphp-buggy-focal
$ lxc shell lp1890263-modphp-buggy-focal
# apt-get update; apt-get -y full-upgrade; apt-get -y autoremove
# reboot
$ lxc shell lp1890263-modphp-buggy-focal
# apt install -y libapache2-mod-php7.4
  The following NEW packages will be installed:
    apache2 ... libapache2-mod-php7.4 ... php7.4-common ...
  ...
# do-release-upgrade -d
  - Answer defaults for everything
  - Allow it to reboot when done, and log back in
$ lxc shell lp1890263-modphp-buggy-focal

# lsb_release -sc
jammy

# apt-cache policy libapache2-mod-php* | grep -B1 Installed:
libapache2-mod-php:
  Installed: 2:8.1+92ubuntu1
--
libapache2-mod-php7.4:
  Installed: 8.1.2-1ubuntu2
--
libapache2-mod-php8.0:
  Installed: (none)
--
libapache2-mod-php8.1:
  Installed: 8.1.2-1ubuntu2
--
libapache2-mod-php5:
  Installed: (none)

# result=$(apt-cache policy libapache2-mod-php8.1 | grep Installed: | cut -d: -f2 | xargs)
# [ "${result}" = "(none)" ] && echo "BUG DETECTED"

# logout

$ lxc stop lp1890263-modphp-buggy-focal
$ lxc delete lp1890263-modphp-buggy-focal

Good job finding a solution that works!

What's the status on Debian accepting something like this?

This also seems like a pattern that would be useful elsewhere. Is it already an established pattern?

I'm reluctant to take this for an SRU without having confidence that we won't end up iterating on it in our stable releases. It feels like it should be well established and socialized first to flush out any issues, and seems odd to me that I've not seen this pattern used before given its wide applicability.

If it's a correct and good solution then I'd expect to be familiar with it. If it's new ground then maybe we should get general acceptance across the archive and in Debian for this pattern before SRUing it.

What am I missing?

I discussed this with other members of the SRU team last week. My notes:

"...that would be a maintenance action being fixed in an SRU, and may pull the rug from under users’ feet by forcibly upgrading them after it has been working on Focal for years. The security implications should be dealt with separately."

An upload of php7.4 to focal-proposed has been rejected from the upload queue for the following reason: "See previous comment".

Thanks everyone for the discussion so far, I'm trying to add a few details and fill a few argumentative gaps.

First let me re-share the simple scenario which I used to illustrate/prove the issues.
Then in another comment below I'll outline a few affects of this overall case.

Example:

On bionic install it "wrong" not using the meta package
$ apt install apache2 libapache2-mod-php7.2
$ do-release-upgrade

Before:
# dpkg -l | grep -e apache -e php
ii apache2 2.4.29-1ubuntu4.23 amd64 Apache HTTP Server
ii apache2-bin 2.4.29-1ubuntu4.23 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.29-1ubuntu4.23 all Apache HTTP Server (common files)
ii apache2-utils 2.4.29-1ubuntu4.23 amd64 Apache HTTP Server (utility programs for web servers)
ii libapache2-mod-php7.2 7.2.24-0ubuntu0.18.04.11 amd64 server-side, HTML-embedded scripting language (Apache 2 module)
ii php-common 1:60ubuntu1 all Common files for PHP packages
ii php7.2-cli 7.2.24-0ubuntu0.18.04.11 amd64 command-line interpreter for the PHP scripting language
ii php7.2-common 7.2.24-0ubuntu0.18.04.11 amd64 documentation, examples and common module for PHP
ii php7.2-json 7.2.24-0ubuntu0.18.04.11 amd64 JSON module for PHP
ii php7.2-opcache 7.2.24-0ubuntu0.18.04.11 amd64 Zend OpCache module for PHP
ii php7.2-readline 7.2.24-0ubuntu0.18.04.11 amd64 readline module for PHP

After:
# dpkg -l | grep -e apache -e php
ii apache2 2.4.41-4ubuntu3.11 amd64 Apache HTTP Server
ii apache2-bin 2.4.41-4ubuntu3.11 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.41-4ubuntu3.11 all Apache HTTP Server (common files)
ii apache2-utils 2.4.41-4ubuntu3.11 amd64 Apache HTTP Server (utility programs for web servers)
ii php-common 2:75 all Common files for PHP packages

There are other variations of the bug, but they all end in a similar situation like this.

And with that situation PHP execution is essentially "off"

Relating this example above to a few details:

#1
Reasonable concern "What if people upgraded, use the (wrong, old and non-serviced) mod-php happily and will now be broken by the SRU?

=> As shown above, once you are in this situation your mod-php is doing nothing.
That IMHO invalidates the regression concern for that set of users.

Only if you would manually ensure to keep all of the bionic php stack (libapache2-mod-php7.2 php7.2-cli php7.2-common php7.2-json php7.2-opcache php7.2-readline) installed from bionic would it still work.
I tried bryce PPA on this special corner case of already a special situation.

Yes it would upgrade (as intended)
The following packages were automatically installed and are no longer required:
  libargon2-0 php7.2-cli php7.2-common php7.2-json php7.2-opcache php7.2-readline
Use 'apt autoremove' to remove them.
The following NEW packages will be installed:
  libapache2-mod-php libapache2-mod-php7.4 php7.4-cli php7.4-common php7.4-json php7.4-opcache php7.4-readline
The following packages will be upgraded:
  libapache2-mod-php7.2
1 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.

PHP is still working, but now in 7.4 instead of the former 7.2.

#2
"Is this just our case, after all one of the server team filed it?"

let me be honest - I'm not a php user or web server admin. But other than this bug we know that "in the wild" this has happened more than a few times.
Bryce has mentioned that while working on this he found a few external references of people hitting and working around it in some way.

#3
Why do we consider this important "now" so late"?

Well, we have to admit of not seeing/knowing it earlier -> that explains the lateness.

I'm (personally, I don't know how bad that really is as I'm neither a PHP nor a security expert) concerned that people might expose secrets by accident.

Imagine the former php code used on bionic was this:

$ echo '<html> <head> <title>Test</title> </head> <body> <?php echo "<p>You only see this</p>"; # secret to DB is foobar ?> </body> </html>' > /var/www/html/index.php

On the web page as served to people on Bionic they saw see:
  "You only see this".
But after upgrading to Focal the bug is active and people will see:
 "You only see this; # secret to DB is foobar ?>"

I know it would be bad practice to put such secrets there in the first place, but still for me this gives the importance of the issue a slight bump.

I'm not sure if this is convincing enough, but I considered it worth to share/summarize.

This isn't Fix Committed; it's awaiting SRU team response again, though with nothing in the queue. We've pinged and added this to a list for SRU team attention out of band.

Waiting for Robie to sync with Steve on a final decision after my clarifications.
We will then follow whatever the decision is.