Complete libvirt migration to Debian style packaging (dependencies, conffiles)

Note also that there is an ubuntu-virt-server package still in the archive which depends on libvirt-bin, despite being labelled transitional, and this is referenced by the virt-host task. This all needs to be sorted before the libvirt-bin binary package could be dropped.

Thank you so much Steve for bringing that up, that is part of the "inherited it that way and never needed to touch" but it clearly should be cleaned up "now" as in "artful" to have a chance to clean it out post 18.04.

There are more dependencies that need to be adapted:
- ubuntu-virt (ubuntu-only)
- kimchi (ubuntu only)
- nova (for nova-compute-libvirt, ubuntu only, need sponsor)
- uvtool (for uvtool-libvirt, git merge proposal to rbasak)

Adding Tasks for these now, and then looking into the mess of old conffiles.

Most of this should have not existed or cleaned up long ago :-/.
But it wasn't, so we now have to decide how to distribute potential fallout the best way for end-users.
We need to "protect" the new files which already exist since yakkety, so we can't just move over the old ones.
There by since in most cases both (old&new) files exist, we can only rm_conffile the no more active ones to ensure users do not accidentially edit those without effect.

Also for SRUs this could cause all sorts of regressions if people fixed up things to properly consider their old configs.
Therefore for now only considering a clean-up in artful and keeping in the back of our mind if e.g. bugs come along like "I change A.conf and nothing happens".
Also the SRU'ability can be decided later on if needed.
I already had noted 79 todo's for the next merge related to cleanup of our Delta, why not a few more around dpkg-maintscript-helper :-/

For now I checked on the files that I found missing (Note: more than initially reported and uncovering potential Debian issues on top):

1. Files Ubuntu failed to properly migrate and now should be removed:
 /etc/init/libvirt-bin.conf
  => upstart gone no more needed

 /etc/dnsmasq.d/libvirt-bin
 /etc/dnsmasq.d-available/libvirt-bin bbf7e62e130a4cb7b6db7c4260883a68 obsolete
  => now /etc/dnsmasq.d-available/libvirt-daemon
     Old file is superfluous X->Y

 /etc/cron.daily/libvirt-bin
  => is now /etc/cron.daily/libvirt-daemon-system
     Old file is superfluous X->Y

 /etc/default/libvirt-bin 619ef67a86531f89f4cf45efde87cb82 obsolete
  => now /etc/default/libvirtd (with changed behavior e.g. no opts set by default as config is set in service file
     Old file has no effect since X->Y

 /etc/apparmor.d/libvirt/TEMPLATE 0d5580a22d95fc622cd5b8efe54b8757 obsolete
  => This was even lost prior to the libvirt-daemon-system changes, when things accepted into Debian but split for lxc/qemu
     Old file has no effect since T->X

2. Files gone, that should actually be there (missing in Debian)
For those I think we want to open bugs in Debian and see if they want to pick them (then we stop missing, but instead move them).
Or if there is an agreeable reason to drop intentionally we shall drop them as well completely.
For now in Ubuntu we "keep" them orphaned until that is clear, in case they come back users won't want their old changes thrown away in between.

 /etc/default/virtlockd
  => missing in Debian as well these days, should be available like virtlogd is
     Note: This is referenced from its service file

 /etc/libvirt/libvirt-admin.conf
  => lost together with /etc/libvirt/libvirt.conf (=1 bug)
     Still built and installed in libvirt make, but not getting through to package, analyze and report

/etc/libvirt/libvirt.conf disappeared from the build log at the place /etc/libvirt/libvirt-admin.conf but is still delivered as part of libvirt-clients. So remove that from the list above that should be reported to Debian.

On /etc/libvirt/libvirt-admin.conf itself is interesting that it seems never shipped with Debian, so it was lost when aligning with Debian again post Xenial.
In particular we can see that Xenial 1.3.1 [1] includes it in the package, but Debian 1.3.1 [2] does not.

[1]: https://launchpadlibrarian.net/319869772/buildlog_ubuntu-xenial-amd64.libvirt_1.3.1-1ubuntu10.10_BUILDING.txt.gz
[2]: https://buildd.debian.org/status/fetch.php?pkg=libvirt&arch=amd64&ver=1.3.1-1&stamp=1453477524&raw=0

Bugs filed at Debian, linking up here

Can only track one as task above, other one is:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863649

There is also a Delta on virt-manager that needs to be changed.
And I realized that this could need a merge anyway - unfortunately I don't know if there just was no time yet or if there were reasons against it.

Kimchi is currently a neglected package in the MOTU space.
The same applies to "Ginger".
It is severely outdated - the one in the archive is from June 2015 (Wily) and since then got no changes at all.
It had upstream a lot of changes/fixes - in general and for Ubuntu in particular.

The recommendation there is to install a .deb file hosted on their website [1].

I'm not deep enough to do the huge merge onto the new version.
And I don't know what is our/their strategic approach to this - but for now I assigned the initial uploader/maintainer.

Old Maintainer was: Frédéric Bonnard <email address hidden>
Looking at the recent packages provided it now is: Aline Manera <email address hidden>

If they are willing to update ginger/kimchi in artful ok - and on the way fix the libvirt-bin dependency can be fixed.
But if not - and the preferred way is to provide the out-of-archive deb then we should remove it from the archive in artful - as-is it is outdated and broken.

[1]: https://kimchi-project.github.io/kimchi/downloads/

There is no Launchpad user for Aline, so I assigned Frederic who did the initial uploads.
Yet given that Aline is listed in the recent .deb packages provided by the kimchi-project I assume she is responsible now.

To reach out more explicitly I mailed them with ubuntu-server on CC [1].

Depending on the feedback there we can decide to drop kimchi/ginger or not.

[1]: https://lists.ubuntu.com/archives/ubuntu-devel/2017-May/039800.html

For Nova I saw 2:16.0.0~b1-0ubuntu4 is already in artful-proposed;
git suggests that 2:16.0.0~b1-0ubuntu5 is still prepared (still at UNRELEASED). So I created a Merge Proposal for git://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/nova into ubuntu5.

[1]: https://code.launchpad.net/~paelzer/ubuntu/+source/nova/+git/nova/+merge/324778

This bug was fixed in the package ubuntu-virt - 1.5

---------------
ubuntu-virt (1.5) artful; urgency=low

  * Update Dependencies to match new libvirt package names (LP: #1694159)

 -- Christian Ehrhardt <email address hidden> Tue, 30 May 2017 07:51:02 +0200

After a few checks and rewrites now there also is a merge proposal for uvtool [1]:

In general I'll assign the bug tasks of nova/uvtool to the receivers of the MPs as "for now" I can only wait in regard to those bug tasks.
Also ubuntu-virt fix is complete, removing assignee there.

[1]: https://code.launchpad.net/~paelzer/uvtool/+git/uvtool/+merge/324779

Brian is already working on the virt-manager merge, so no need to interrupt all the work he has done - see bug 1667114.
Which makes the virt-manager we consider here just the dependency change + a few related changes in help texts that Ubuntu adds.

Tested the virt-manager changes:
- Automated testing is only minimal but was working.
- Manual tests with the updated version worked as before
- only effect that surfaced from the change to me as a "user" was that libvirt-bin is now auto-removable (which is just right)
Note: I also updated the merge bug to make sure things are not lost on the merge.

That said the virt-manager change looks good and can be pushed to artful IMHO.

This bug was fixed in the package virt-manager - 1:1.3.2-3ubuntu5

---------------
virt-manager (1:1.3.2-3ubuntu5) artful; urgency=medium

  * Update Dependencies to match new libvirt package names (LP: #1694159)
    This aligns with Debian and effectively drops the former change:
    "Change Recommends to libvirt-bin instead of libvirt-daemon-system
    which isn't available in Ubuntu" and modifies a few texts the ubuntu
    delta to match.
    - debian/control: drop dependency change
    - debian/patches/more_helpful_error_message.patch: help text update
    - debian/patches/use_ubuntu_package_names.patch: help text update

 -- Christian Ehrhardt <email address hidden> Tue, 30 May 2017 13:25:02 +0200

I got a reply on my mail which kind of echoes my sentiment for kimchi/ginger.
It referred to a similar ML post at [1].
+1 for removing them, but we have to give the owners a chance to comment.

[1]: https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2016-December/017231.html

I was summarizing the cases we have into three categories:

Old Package A
New Packages B, C

Certain kinds of conffile transitions that forgot to properly mv_conffile/rm_conffile
Naturally new installs are fine, but we have upgraded systems facing the following cases.

Syntax
<PKG>-<Conffile> -> change that happened

1. A-1 -> no more needed
2. A-2 -> B-2 "same content" but different filename, upgraded systems have A-2 && B-2 on disk
3. A-3 -> B-3, B-3 is somewhat related, but of different content than A-3 (e.g. needed different defaults)

I think the following is the right action:
1. rm_conffile
2. Correct would have been mv_conffile initially, but it was not what happened.
   Now we have multiple types of users:
   2.1 had changes to A-2 that are no more active since yakkety or utopic - if it wasn't a problem so far it won't be too much of a problem to remove it now. (rm_conffile)
   2.2 had no changes to A-2, the defaults of B-2 should apply (rm_conffile)
   2.3 already have changes to B-2, whatever is in A-2 should not break that (rm_conffile)
   2.4 had changes to A-2 and now have changes to B-2. Some special case of 2.1, but essentially they are not broken or broken since yakkety so wile not nice (rm_conffile)
3. Correct would have been mv_conffile initially, but it was not what happened.
   Even less than with #2 we can retain the old content, since it would break things.
   All arguments of #2 apply plus the "need" to get the old defaults changed (rm_conffile)

I wonder if for #2 and less likely for #3 we would want some magic of "If A-2 was not the original A-2, then take over the content to B-2 IFF B-2 is the default B-2". Yet I think in this case the upgrade would be a behavior change that potentially is not wanted. If users "cared" about their change to A-2 they already suffered and implemented it in B-2.

Discussed with Rbasak and we agreed while one could try to do some magic, the chance to do it wrong is bigger than the gain.
Also we have two kinds of users, those who are already broken and we "can't safe", and OTOH those who will go LTS->LTS which we can actually save.
So for those who will do LTS->LTS (Which later also applies to Ubuntu Cloud Archive if you go Xenial -> >=Pike) - there we might be able to improve the situation.
Because it turned out while some of those files were just forgotten - others should have been handled but that didn't work.

libvirt-daemon-system.maintscript has entries for some of the files, but that did not work.
I'm now debugging into that and maybe we will end up with:
1. rm-conffiles if coming from the broken releases (versions in yakkety/zesty)
2. mv_conffiles for those coming from Xenial

Debugging deeper into this now which will take a while ...

Note: potentially similar issue in squid/squid3/squid transition https://git.launchpad.net/~racb/ubuntu/+source/squid3/commit/?id=bb2787bb2c0a0f3efa3d660299dd41d9e6716c57#

Just to be sure I checked e.g. https://launchpadlibrarian.net/317266660/buildlog_ubuntu-artful-amd64.libvirt_2.5.0-3ubuntu7_BUILDING.txt.gz that package holds none of those conffiles - so all that is still listed on it is broken conffiles of some sort.

Both contributions to Debian seem to be accepted.
We for now can expect the following files to "come back":
- /etc/default/virtlockd
- /etc/libvirt/libvirt-admin.conf

Those had no name change, just package ownership changed - so no rm_conffile or such needs to be added.
It will need Debian to upload (after release+un-freeze?) and us to merge that.
I've made notes at the merge doc to check that on the merge.

Existing rules that seemed to not fully work in the past related to our reports:
  # on Steve and paelzers system
  mv_conffile /etc/default/libvirt-bin /etc/default/libvirtd 1.3.3-2 libvirt-bin
  mv_conffile /etc/init.d/libvirt-bin /etc/init.d/libvirtd 1.3.3-2 libvirt-bin
  # on Steves system
  rm_conffile /etc/apparmor.d/libvirt/TEMPLATE 1.3.3-2 libvirt-bin
  rm_conffile /etc/apparmor.d/libvirt/TEMPLATE 1.2.7-5~ libvirt-daemon-system
  rm_conffile /etc/apparmor.d/libvirtd/TEMPLATE 1.2.7-5~ libvirt-bin
  rm_conffile /etc/apparmor.d/libvirtd/TEMPLATE 1.2.7-5~ libvirt-daemon-system

Now neither of our systems can be considered the cleanest systems ever so I'll work on and attach a clean repro next.

Theories to debug:
- Maybe because libvirt-bin was a transitional package and "has no maintainter scripts"?
- In the squid patch linked above it is suggested that dpkg-maintscript-helper has certain constraints on package owning/existing to trigger
- ...

Shortest test (WIP), it shows:
- Several dir deletes can't happen due to not-empty directories.
  Those are created later on by libvirt and the packages don't know them, those are all taken over so "ok".
    dpkg: warning: unable to delete old directory '/etc/libvirt/nwfilter': Directory not empty
    [...]
- Other dirs fail to delete due to stuff we forgot.
  We will handle those with rm_conffiles, but in all cases the new packages will extract things there so the dir won't (and shouldn't go away)
    dpkg: warning: unable to delete old directory '/etc/dnsmasq.d-available': Directory not empty
    [...]
- some valid handling of conffiles
  Installing new version of config file /etc/libvirt/libvirtd.conf ...
  [...]
- dpkg --status libvirt-bin listing way too much as obsolete.
  The status on libvirt-daemon-system lists them as well.
  dpkg -S Seems aware of that and lists libvirt-daemon-system is the sole owner
- some of the mv_conffiles in the package clearly did not work
  $ lxc exec ${REL}-test-virtdeps-X-to-UCA-O -- dpkg -S /etc/default/libvirt-bin /etc/default/libvirtd
    libvirt-bin: /etc/default/libvirt-bin
    libvirt-daemon-system: /etc/default/libvirtd
- dpkg -L more clearly pointing to those that are a problem

The list of dpkg -L libvirt-bin seems to be the "more trustworthy" one.
Here all of them categorized as later reference:

Since Steve also had a lost TEMPLATE which is older, so I also checked for T->X->UCA-O and could reproduce. From there we have an additional rm_conffile that did not work.

Forgotten to remove:
- /etc/dnsmasq.d-available/libvirt-bin
- /etc/cron.daily/libvirt-bin
- /etc/init/libvirt-bin.conf

mv_conffile failed to work
- /etc/default/libvirt-bin

rm_conffile failed to work
- /etc/apparmor.d/libvirt/TEMPLATE

Lost in Debian (will be readded)
- /etc/default/virtlockd
- /etc/libvirt/libvirt-admin.conf

The generated line in /var/lib/dpkg/info/libvirt-daemon-system.postinst for the example of /etc/default/libvirt-bin is:
dpkg-maintscript-helper mv_conffile /etc/default/libvirt-bin /etc/default/libvirtd 1.3.3-2 libvirt-bin -- "$@"

But executed on the target as-is it works:
DPKG_MAINTSCRIPT_NAME=postinst dpkg-maintscript-helper mv_conffile /etc/default/libvirt-bin /etc/default/libvirtd 1.3.3-2 libvirt-bin -- configure 1.3.1-1ubuntu10.10

Continue debugging ...

Interrupt on the kimchi Discussion from the Mail thread - quoting Frederic who uploaded what we have atm.

"Hi Christian,
I mainly worked on packaging kimchi/ginger and the other ones after the
project split the source in other components (wok, ginger-base in
addition to kimchi and ginger) so I'm not part of the
development team and I then transfered that knowledge to a member of the
kimchi team who was supposed to pursue the packaging for kimchi 2 but
that packaging effort was discontinued as stated in the last comments of
those debian ITP bugs : 847102 847105 847108 772823 775747 846595 846598
Last update for Ubuntu was in the PPA :
https://launchpad.net/~ibmpackages/+archive/ubuntu/kimchi-project/
before Lucio Correia announced the end of the packaging effort.

I feel like the project is not [very] active anymore :
http://kimchi-project.github.io/kimchi/meetings/
http://lists.ovirt.org/pipermail/kimchi-devel/
https://github.com/kimchi-project/kimchi/commits/master

Several people asked me questions about the .deb etc and I had no
other answer than "no support from upstream anymore, no news either".
So I guess removing may be the only solution. It would be nice to have
some feedback from upstream before that though."

That is another +0.75 to remove it.
Waiting a bit more if others chime in.

The rm_conffile was also working manually but not on the upgrade itself.

So I was running with DPDK_DEBUG=1 for dpkg-maintscrip-helper in the real upgrades:

Preparing to unpack .../libvirt-bin_2.5.0-3ubuntu5~cloud0_amd64.deb ...
Unpacking libvirt-bin (2.5.0-3ubuntu5~cloud0) over (1.3.1-1ubuntu10.10) ...
[...]
Selecting previously unselected package libvirt-daemon-system.
Preparing to unpack .../libvirt-daemon-system_2.5.0-3ubuntu5~cloud0_amd64.deb ...
DEBUG: dpkg-maintscript-helper: Executing /usr/bin/dpkg-maintscript-helper mv_conffile in preinst of libvirt-daemon-system
DEBUG: dpkg-maintscript-helper: CONFFILE=/etc/default/libvirt-bin -> /etc/default/libvirtd PACKAGE=libvirt-bin LASTVERSION=1.3.3-2 ACTION=install PARAM=

Look at the PARAM= argument.
Since libvirt-daemon-system is new it has no old version.
That is why all the commands fail on the real upgrade.

That sounds more than ever like the squid change we had linked above, implementing that and
 adding the missed rm_conffiles for a test.

Moving those calls to the right package seems to be an easier fix than the referred hack in preint/postinst - I have a ppa building and testing.

That looks much better now:
Preparing to unpack .../libvirt-bin_2.5.0-3ubuntu8~ppa2_amd64.deb ...
DEBUG: dpkg-maintscript-helper: Executing /usr/bin/dpkg-maintscript-helper mv_conffile in preinst of libvirt-bin
DEBUG: dpkg-maintscript-helper: CONFFILE=/etc/default/libvirt-bin -> /etc/default/libvirtd PACKAGE=libvirt-bin LASTVERSION=2.1.0-1ubuntu9~ ACTION=upgrade PARAM=2.5.0-3ubuntu5~cloud0

Initial sniff tests were successful, now preparing various upgrade vectors to verify the result is the way we want it.

Final nail in the coffing from the second deb packager of kimchi/ginger on the mail thread [1].
(Replies might be stuck in moderation I guess, that is why I quote them)

"It would be great to have Kimchi and Ginger updated on Ubuntu repositories.
But after so many tries (from Fred and Lucio) and no Debian developer willing to sponsor the package update on Debian community, we gave up on that as the development team reduced a lot due IBM strategy changes.
If some of you want to sponsor that update on Debian community, I am more than happy to keep it going. Otherwise, I agree it is better to remove the package than have them broken there."

[1]: https://lists.ubuntu.com/archives/ubuntu-devel/2017-May/039800.html

@~ubuntu-archive
per comments #8 #16 #25 and #29 I think we reached consensus and should remove the following packages from the archive in Artful:
- kimchi
- ginger

Reasoning:
- broken
- outdated
- not maintained
- no further dependencies on them that would block

Looking for an archive admin that is willing to consider removing them.
Unassigning from Frederic and subscribing ~ubuntu-archive so one can pick

FYI - For users who come here "missing" these packages (well likely missing a working version).
The projects strategy itself seems to be self provided deb's - while I don't like that you can get it at [1].
Furthermore there is a ppa maintained by IBM which you could use at [2].
But also please read the referred comments and links if you really want that.

[1]: https://kimchi-project.github.io/kimchi/downloads/
[2]: https://launchpad.net/~ibmpackages/+archive/ubuntu/kimchi-project/

It turns out mv_conffile + package rename doesn't work correctly.
It would not realize there was a changed conffile and drop the old content completely.
It would neither retain in moved file, nor in dpkg-bak which is the worst case.

Furthermore and more important is that the the combination of
  mv A B oldver
  rm B newver
doesn't work.
This is needed to mv-retain data on LTS->LTS upgrade which "can still be saved".
While at the same time drop obsolete conffiles (including retain custom data in backups)
But instead the preinst of the RM will make the conffile unavailable for the postinst of the MV.
The postinst of RM will then remove it.
So that is equivalent to "just" RM.
Note: if MV finds an equal md5sum it is actually a RM in via .dpkg-remove
Overview:
          diff default
pre mv - does nothing mv .dpkg-remove
pre rm - mv .dpkg-backup mv .dpkg-remove
post rm - would mv, but named .dpkg-backup now rm .dpkg-remove
post rm - .dpkg-backup to dpkg-bak rm .dpkg-remove

The conffiles that changed were those that almost never have user changes, still we want to make it correct. So the implmentation tries to move old changes where applicable, but in a few cases only creates backups to not interfere too much.
So likle 99% of the cases there won't be changes anyway and it will just be deletes to clean up.

It turns out there also was a non packaged symlink (created in maintainer scripts) that had to be taken care of.

So we need another tweak which - just in the critical combination fixes
up the "state machine" to work with install/upgrade/abort but also both combinations of RM/MV as needed.
To do so if in these special "version window" in preinst after DEBHELPER we need to undo the RM (only if happened which matches that there were custom changes, but keep it in all other cases)

Finally the changes in /etc/init.d/libvirt-bin are critical throughout the update, we don't want those to be moved. Through the combo of renaming plus package change that would cause issues with insserv (or need even more hacky workarounds). Therefore for that we only want to create the .dpkg-bak and message if it was modified (unlikely anyway).

PPA Fixes (https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2784/+packages):
FixX: is a build for Xenial to similulate an upgrade LTS->LTS with the fix.
FixA: is a fixed version for Artful to check full upgrade paths, as well as e.g. Zesty->Fix (not having old files)

0. Calls
./conffiles-forgotten-test.sh TXUF
./conffiles-forgotten-test.sh TXF
./conffiles-forgotten-test.sh ZAF
./conffiles-forgotten-test.sh TXUF custom
./conffiles-forgotten-test.sh TXF custom

1. Upgrade paths:
 1.1 T->X->UCA-O-FixX (like full upgrade through Y,Z,A but faster)
 1.2 T->X->FixX (simulating LTS->LTS)
 1.3 Z->FixA (no old things around, shall not fail on upgrades on any of our code)

2. Default conffiles
 2.1 if all files are on default content only, they are deleted and not retained
 2.2 T-X-Fix e.g. default /etc/default/libvirt-bin not interferring with new /etc/default/libvirtd content
 2.3 T-X-UCA-Fix default /etc/default/libvirt-bin (duplicate) not interferring with new /etc/default/libvirtd content
 2.4 /etc/default/virtlockd and /etc/libvirt/libvirt-admin.conf shall still be around as they work and will come back via Debian
 2.5 remove default /etc/dnsmasq.d/libvirt-bin link

3. custom conffiles
 3.1 T-X-Fix custom data on TEMPLATE be removed, but retained in backup
 3.2 T-X-Fix custom data on /etc/cron.daily/libvirt-bin moved
 3.3 T-X-Fix custom data on /etc/default/libvirt-bin moved
 3.4 T-X-Fix custom data on /etc/init.d/libvirt-bin moved
 3.5 T-X-Fix Leave custom /etc/dnsmasq.d/libvirt-bin link untouched
 3.6 T-X-UCA-Fix custom data on TEMPLATE removed, but retained in backup
 3.7 T-X-UCA-Fix custom /etc/default/libvirt-bin (duplicate) moved, new data retained .dpkg-new
 3.8 T-X-UCA-Fix custom data on /etc/cron.daily/libvirt-bin (duplicate) moved, new data retained .dpkg-new
 3.9 T-X-UCA-Fix leave custom /etc/dnsmasq.d/libvirt-bin link untouched
 3.10 T-X-UCA-Fix custom /etc/init.d/libvirt-bin (want change) removed old, retained in dpkg-bak

From what I saw I think all planned use-cases and tests were correct now.
Attaching the logs.

But since it is weekend and we are sprinting next week I'm not uploading now and run away.
Instead I might check the logs more in depth some-when on the trip or sprint - maybe even catch somebody to review there and then upload.

Pushed the unreleased preliminary changes to [1].

[1]: https://git.launchpad.net/~libvirt-maintainers/ubuntu/+source/libvirt/log/?h=ubuntu/artful&showmsg=1

Found a test issue in the TXUF cases which need to be re-ran, also we want to add cases with conflicting new content.

4. conflicting custom duplicate conffiles
 4.1 T-X-UCA-Fix custom /etc/default/libvirt-bin (rm + retain old duplicate)
 4.2 T-X-UCA-Fix custom data on /etc/cron.daily/libvirt-bin (rm + retain old duplicate)
 4.4 T-X-UCA-Fix custom /etc/init.d/libvirt-bin (want change in new file)

After some cleanups things make more sense now.
For >=Yakkety upgraders we really only want to clean up and not mess with their setup.
So the design slightly changed to do so as intended.
Otherwise we might "bring back" old changes breaking a working system.
Instead the remove + message will help to
a) clean up
b) remind the user if he wants to re-setup anything

Also the tests are improved and have a new case now.

0. Calls
./conffiles-forgotten-test.sh TXUF
./conffiles-forgotten-test.sh TXF
./conffiles-forgotten-test.sh ZAF
./conffiles-forgotten-test.sh TXUF custom
./conffiles-forgotten-test.sh TXF custom
./conffiles-forgotten-test.sh TXUF newcustom

1. Upgrade paths:
 1.1 T->X->UCA-O-FixX (like full upgrade through Y,Z,A but faster)
 1.2 T->X->FixX (simulating LTS->LTS)
 1.3 Z->FixA (no old things around, shall not fail on upgrades on any of our code)

2. Default conffiles
 2.1 if all files are on default content only, they are deleted and not retained
 2.2 T-X-Fix e.g. default /etc/default/libvirt-bin not interferring with new /etc/default/libvirtd content
 2.3 T-X-UCA-Fix default /etc/default/libvirt-bin (duplicate) not interferring with new /etc/default/libvirtd content
 2.4 /etc/default/virtlockd and /etc/libvirt/libvirt-admin.conf shall still be around as they work and will come back via Debian
 2.5 remove default /etc/dnsmasq.d/libvirt-bin link

3. custom conffiles
 3.1 T-X-Fix custom data on TEMPLATE be removed, but retained in backup
 3.2 T-X-Fix custom data on /etc/cron.daily/libvirt-bin moved
 3.3 T-X-Fix custom data on /etc/default/libvirt-bin moved
 3.4 T-X-Fix custom data on /etc/init.d/libvirt-bin moved
 3.5 T-X-Fix Leave custom /etc/dnsmasq.d/libvirt-bin link untouched
 3.6 T-X-UCA-Fix custom data on TEMPLATE removed, but retained in backup
 3.7 T-X-UCA-Fix custom /etc/default/libvirt-bin (duplicate) removed, but retained in backup
 3.8 T-X-UCA-Fix custom data on /etc/cron.daily/libvirt-bin removed, but retained in backup
 3.9 T-X-UCA-Fix leave custom /etc/dnsmasq.d/libvirt-bin link untouched
 3.10 T-X-UCA-Fix custom /etc/init.d/libvirt-bin (want change) that was moved on ->yakkety, so it should just not be around

4. conflicting custom duplicate conffiles that "should have been" moved before
 4.1 T-X-UCA-Fix custom /etc/default/libvirt-bin removed, but retained in backup, not clashing new content
 4.2 T-X-UCA-Fix custom data on /etc/cron.daily/libvirt-bin removed, but retained in backup, not clashing new content
 4.3 T-X-UCA-Fix custom /etc/init.d/libvirt-bin that was moved on ->yakkety, new content in libvirtd should be unaffected

On the release notes for Artful we should add a about this along the usual Libvirt update info.
There is no release note wiki/template for Artful yet, so I added a tracker for that.

An example could look like:

There was an issue in libvirt (bug 1694159) about handling conffiles when upgrading from Xenial (16.04) to latter releases. This issue lead to duplicate conffiles carrying the old libvirt-bin names/references.
Since Yakkety (16.10) those files are inactive but still around. That is misleading at best - e.g. if users try to modify them and wonder why the daemon do not pick up the changes. Therefore those conffiles will get cleaned up now.
For LTS (16.04) -> LTS (18.04) upgraders, but also for Xenial -> Ubuntu Cloud Archive Pike transitions this will be fine. In those cases there never were inactive duplicates, so the upgrade will move and retain user configurations as it should have done back in Yakkety.
But users that are already on Yakkety or later will have the duplicate conffiles around and moving those might negatively affect a currently working setup. Therefore in those cases the old conffiles are removed. If there was any custom content present they will be backed up as usual with a .dpkg-bak suffix and the user prompted about that on the upgrade.
In general changes to the affected conffiles are rather rare, so for most users the upgrade will just remove a few old inactive conffiles.

This bug was fixed in the package libvirt - 2.5.0-3ubuntu8

---------------
libvirt (2.5.0-3ubuntu8) artful; urgency=medium

  * fix conffile upgrade handling to avoid obsolete files
    and inactive duplicates (LP: #1694159)
    - d/libvirt-daemon-system.maintscript: revert to Debian content
    - d/libvirt-bin.maintscript: add missing rm_conffile related to
      dropping upstart.
    - d/libvirt-bin.maintscript: add missing rm of conffiles due
      to re-aligning with debian package names since yakkety.
    - d/libvirt-bin.maintscript: for LTS->LTS upgraders try to move and retain
      custom changes.
    - d/libvirt-bin.maintscript: for upgraders from yakkety or later remove
      the (now duplicate) conffiles, but retain custom changes in backups if
      they exist
    - d/libvirt-bin.preinst: drop manual mv of conffiles which lacked
      retaining changes and upgrade-abort handling.
    - d/libvirt-bin.preinst: handle upgrades up to the latest predecessor
      possible before yakkety.
    - d/libvirt-bin.preinst: fixup the combination of rm+mv conffile in case
      the package is upgrading from pre yakkety.
    - d/libvirt-daemon-system.postinst: clean up old dnsmasq enablement symlink
      if unmodified.

 -- Christian Ehrhardt <email address hidden> Wed, 31 May 2017 14:29:51 +0200

$ remove-package -m 'Outdated, unmaintained; LP: #1694159' kimchi ginger
Removing packages from artful:
 kimchi 1.5.0-0ubuntu1 in artful
  kimchi 1.5.0-0ubuntu1 in artful amd64
  kimchi 1.5.0-0ubuntu1 in artful arm64
  kimchi 1.5.0-0ubuntu1 in artful armhf
  kimchi 1.5.0-0ubuntu1 in artful i386
  kimchi 1.5.0-0ubuntu1 in artful ppc64el
  kimchi 1.5.0-0ubuntu1 in artful s390x
 ginger 1.3.0.1-0ubuntu1 in artful
  ginger 1.3.0.1-0ubuntu1 in artful amd64
  ginger 1.3.0.1-0ubuntu1 in artful arm64
  ginger 1.3.0.1-0ubuntu1 in artful armhf
  ginger 1.3.0.1-0ubuntu1 in artful i386
  ginger 1.3.0.1-0ubuntu1 in artful ppc64el
  ginger 1.3.0.1-0ubuntu1 in artful s390x
Comment: Outdated, unmaintained; LP: #1694159
Remove [y|N]? y
2 packages successfully removed.

Fixes needed in Debian went into 3.5.0-1 - so the merge will pick them up and the few remaining obsolete files will be correctly part of the pkg now.

This bug was fixed in the package uvtool - 0~git136-0ubuntu1

---------------
uvtool (0~git136-0ubuntu1) bionic; urgency=medium

  * New upstream snapshot: ship tested templates for some non-Intel
    architectures and use them automatically when on those platforms.
    Thanks to Christian Ehrhardt. LP: #1452016.

 -- Robie Basak <email address hidden> Thu, 07 Dec 2017 10:26:51 +0000

This bug was fixed in the package nova - 2:17.0.0~b3-0ubuntu3

---------------
nova (2:17.0.0~b3-0ubuntu3) bionic; urgency=medium

  * d/control: Drop min version of python-sphinxcontrib.blockdiag/seqdiag.

 -- Corey Bryant <email address hidden> Thu, 01 Feb 2018 09:30:56 -0500