New upstream microreleases 9.3.16, 9.5.6

For Zesty things are more complex as we need to get the 9.6 release out of proposed first.
But doing SRUs to released versions now in time to get users updated.

:-/ I can't create ack/nack tasks for this myself, it should be
9.3 - Trusty
9.5 - Xenial
9.5 - Yakkety
9.5 - Zesty

Working build and autopkgtests can be found in the following Bileto tickets:
https://bileto.ubuntu.com/#/ticket/2468
https://bileto.ubuntu.com/#/ticket/2469
https://bileto.ubuntu.com/#/ticket/2470

I can't sponsor the uploads, maybe one of the SRU Team with the right powers can just copy from Bileto to $release-proposed - not sure on that.

Surely one can get the dsc and tarballs (needed since orig tarball changed) from the ppas if needed.

I attach the debdiffs later as well.
Now waiting for the tests to complete and then maybe sync with Pitti on the Results and if he can spare the time a review on the diffs.

I realized that the diffs are also reachable via bileto as it generated them, so no need to upload equivalent debdiffs unless requested.

For Trusty I see https://bileto.ubuntu.com/excuses/2468/trusty.html
That is all completing, but postgresql-9.5/armhf.
Quoting Pitti fromt he last stable update "These are fallout from moving from LXC to LXD. For devel I fixed the latter in https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-common.git/commit/?id=fc40fc34ce -- functionally everything is correct, the test just failed on unexpected stderr output from a warning in netstat."

For Xenial I see https://bileto.ubuntu.com/excuses/2469/xenial.html
That is again as above, only the lxd based environments.
s390x was not available pre xenial, so it fails now but not on trusty - as arm this is a lxd based test environment - so it should be the same issue - correct?

Yakkety seems worse thou https://bileto.ubuntu.com/excuses/2470/yakkety.html
we have:
1. the same LXD based issue as Xenial/Trusty

2. pg-repack - that seems a nova issue at first
"novaclient.exceptions.OverLimit: Quota exceeded for ram: Requested 1536, but already used 50688 of 51200"
But it recovers from that and then runs on all arches into "test tablespace ... FAILED"
That LGTM before http://autopkgtest.ubuntu.com/packages/pg-repack/yakkety/amd64
I don't know - is that known to be flaky?
I can't retrigger the test lacking the matching upload rights :-/

3. postfix failing on amd64 and s390x
Those seem to be caused by connectivity issues, again I'm not sure if those are known and/or transient only. The history e.g. on http://autopkgtest.ubuntu.com/packages/postfix/yakkety/amd64 suggests that they might be just flaky as well.

Ultimately I'd like to let that go to the judgement of Pitti (Pinging him on IRC)
He can also trigger the retest if that is what we want/need.

To easen referencing - the last update was tracked in bug 1637236

Debdiffs LGTM. For the tests: postfix tests are a bit flaky. But the pg_repack regression should be investigated more seriously, that doesn't look like a flake at all (it also happened on all architectures):

2017-02-14 11:07:37 UTC [2371-1] ubuntu@contrib_regression ERROR: could not create unique index "idx_badindex_n"
2017-02-14 11:07:37 UTC [2371-2] ubuntu@contrib_regression DETAIL: Key (n)=(10) is duplicated.

That seems very much related to the prominent fix in that release:

+ - Fix a race condition that could cause indexes built with CREATE INDEX
+ CONCURRENTLY to be corrupt (Pavan Deolasee, Tom Lane).
+ If CREATE INDEX CONCURRENTLY was used to build an index that depends on
+ a column not previously indexed, then rows inserted or updated by
+ transactions that ran concurrently with the CREATE INDEX command could
+ have received incorrect index entries.

I suggest asking on https://www.postgresql.org/list/pgsql-bugs/ or that packager ML you recently joined.

Oh, I forgot: This might very well be a bug in pg_repack's tests too of course.

Weird/sorry that your first update causes such trouble.. :-/

No need to excuse, murphy is everywhere :-)

I'll debug as far as I get and call out to the pkg list then.
I already confirmed that Debian is also affected, so we might as well work together on the solution. Yet I want to spend a bit time debugging myself to close my skill gap in that regard before calling out to them :-)

Jeremy Bicha spotted that in debici 2 days ago and I chimed in there, linking the Debian bug for what we see in the Yakkety case.

I found the issue in pg-repack and suggested a fix upstream as well as to Debian.

Upstream PR https://github.com/reorg/pg_repack/pull/115

Added a pg-repack task, this has to be fixed (SRU) for Yakkety before the MRE for Yakkety will migrate.

It is only a test case failure (no real problem), so if the SRU Team would consider it as is (and can bypass the failing dep8 test) we could unblock the MRE and upload that now to proposed.

For Zesty things are more complicated as there the postgres-9.6 in proposed (which shall go away as z should stick to 9.5) makes things more complex.

That said I think the following steps are:
1. Fixed pg-regress into Yakkety
2. MRE into T/X/Y

I'll provide a debdiff for #1 as I can't upload for pg-regress.

Giving things 24 hours to let Debian and/or upstream respond on the suggestion.
Preparing an SRU Teamplate for the pg-repack fix already.

@nacc / @pitti - let me know what you think on this finding and if we should go this way.

FTR, I see no reason to block the update on the pg-repack issue as that's a test suite bug only. Nice job tracking this down!

Attaching debdiffs in cae copy from Bileto (or debdiff from there) won't work

Attaching debdiffs in case copy from Bileto (or debdiff from there) won't work

Attaching debdiffs in case copy from Bileto (or debdiff from there) won't work

I agree Martin - if ok with the SRU Team - there is no reason to block on the tests.
Maybe we should also retrigger the Yakkety postfix tests to prove that they recover and were just flaky as we assumed.
I pinged nacc to do so.

So the next steps will need at least someone with upload rights and eventually the the SRU Team (or SRU Team member can do all steps).

0. evaluate all the changes from your POV
1. move forward the MRE release into T/X/Y release
   - either from Bileto if that works
   - or by responsoring my diffs (I attached debdiffs just to be sure in case it is needed)
2. SRU approve the MREs for T/X/Y
3. review, sponsor and SRU approve the pg-repack change

I published the packages from bileto and accepted them into -proposed. Now let the distro CI do its spiel and please flip to v-done once the tests are done.

A few of the flaky tests show up as expected, and one that might need an extra look at "mimeo" fails on arm due to newpid.
Nish and I will (re-)test some and if needed evaluate that and come back here with a v-done after that.

So with pg-repack understood and fix on its way (no need to block the postgres transition on it) the only thing left is mimeo.

Mimeo fails only on armhf for a dependency issue by failing to set up newpid correctly:
Setting up newpid (7) ...
Failed to set capabilities on file `/usr/bin/newpid' (Operation not permitted)
The value of the capability argument is not permitted for a file. Or the file is not a regular (non-symlink) file
dpkg: error processing package newpid (--configure):
 subprocess installed post-installation script returned error exit status 1

That happens for Xenial & Yakkety but only armhf.
Trusty has no mimeo package.

Local (x86) repro works, just as the official x86 tests do.

At least the autopkglogs suggest some relation to the stable update:
http://autopkgtest.ubuntu.com/packages/mimeo/xenial/armhf
http://autopkgtest.ubuntu.com/packages/mimeo/yakkety/armhf
And I mean the LAST one - not this one.

Since 9.5.5 this seems broken, but we are here at 9.5.6.

That is again quoting Pitti:
"All pass except for mimeo/armhf and postgresql-9.5/armhf. These are fallout from moving from LXC to LXD. For devel I fixed the latter in https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-common.git/commit/?id=fc40fc34ce -- functionally everything is correct, the test just failed on unexpected stderr output from a warning in netstat."

So we have all issues remaining in autopkgtests understood, manual tests are good and at least for some of the dep8 tests fixes are already on the way. I'm assigning the pg-repack to myself to not forget to drive this.

Setting verification-done - please migrate according to former comments.

The verification of the Stable Release Update for postgresql-9.3 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package postgresql-9.5 - 9.5.6-0ubuntu0.16.04

---------------
postgresql-9.5 (9.5.6-0ubuntu0.16.04) xenial; urgency=medium

  * New upstream release (LP: #1664478)
    - Fix a race condition that could cause indexes built with CREATE INDEX
      CONCURRENTLY to be corrupt (Pavan Deolasee, Tom Lane).
      If CREATE INDEX CONCURRENTLY was used to build an index that depends on
      a column not previously indexed, then rows inserted or updated by
      transactions that ran concurrently with the CREATE INDEX command could
      have received incorrect index entries. If you suspect this may have
      happened, the most reliable solution is to rebuild affected indexes
      after installing this update

    - Details about other changes:
      http://www.postgresql.org/docs/9.5/static/release-9-5-6.html

 -- Christian Ehrhardt <email address hidden> Tue, 14 Feb 2017 09:29:09 +0100

This bug was fixed in the package postgresql-9.5 - 9.5.6-0ubuntu0.16.10

---------------
postgresql-9.5 (9.5.6-0ubuntu0.16.10) yakkety; urgency=medium

  * New upstream release (LP: #1664478)
    - Fix a race condition that could cause indexes built with CREATE INDEX
      CONCURRENTLY to be corrupt (Pavan Deolasee, Tom Lane).
      If CREATE INDEX CONCURRENTLY was used to build an index that depends on
      a column not previously indexed, then rows inserted or updated by
      transactions that ran concurrently with the CREATE INDEX command could
      have received incorrect index entries. If you suspect this may have
      happened, the most reliable solution is to rebuild affected indexes
      after installing this update

    - Details about other changes:
      http://www.postgresql.org/docs/9.5/static/release-9-5-6.html

 -- Christian Ehrhardt <email address hidden> Tue, 14 Feb 2017 09:28:18 +0100

This bug was fixed in the package postgresql-9.3 - 9.3.16-0ubuntu0.14.04

---------------
postgresql-9.3 (9.3.16-0ubuntu0.14.04) trusty; urgency=medium

  * New upstream release (LP: #1664478)
    - Fix a race condition that could cause indexes built with CREATE INDEX
      CONCURRENTLY to be corrupt (Pavan Deolasee, Tom Lane).
      If CREATE INDEX CONCURRENTLY was used to build an index that depends on
      a column not previously indexed, then rows inserted or updated by
      transactions that ran concurrently with the CREATE INDEX command could
      have received incorrect index entries. If you suspect this may have
      happened, the most reliable solution is to rebuild affected indexes
      after installing this update

    - Details about other changes:
      https://www.postgresql.org/docs/9.3/static/release-9-3-16.html

 -- Christian Ehrhardt <email address hidden> Tue, 14 Feb 2017 09:29:18 +0100

FYI - Upstream PR for pg-repack got an ack.
https://github.com/reorg/pg_repack/pull/115

Also an ack from Debian and a release into Debian unstable.
Could one look into sponsoring fix-1664478-pg-repack-testcase.debdiff into Yakkety (MOTU).

Hello ChristianEhrhardt, or anyone else affected,

Accepted pg-repack into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pg-repack/1.3.4-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Thank you Brian,
I checked excuses and can confirm that the fix works getting the Test to work with the new postgres MRE release.

E.g. https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-yakkety/yakkety/amd64/p/pg-repack/20170223_190413_6d78e@/log.gz
From todays http://people.canonical.com/~ubuntu-archive/proposed-migration/yakkety/update_excuses.html#pg-repack

That shows the test running correctly against 9.5.6 which formerly failed.
All arched look like http://autopkgtest.ubuntu.com/packages/pg-repack/yakkety/amd64

Which means broken by the MRE as analyzed in this bug, but good now.

That said, verification done.

Also zesty has synced latest Debian which has my fix, so setting pg-repack fixed in devel as well.

postgresql-9.6 has migrated out of zesty-proposed and postgresql-9.5 has been removed from zesty.

This bug was fixed in the package pg-repack - 1.3.4-1ubuntu0.1

---------------
pg-repack (1.3.4-1ubuntu0.1) yakkety; urgency=medium

  * Extend testcase to work with latest microrelease (LP: #1664478)

 -- Christian Ehrhardt <email address hidden> Wed, 15 Feb 2017 11:23:42 +0100