[UBUNTU 20.04] Vsock can't be used with Secure Execution, required argument not supported

Please add a reference (or references) to the upstream fix(es).

------- Comment From <email address hidden> 2021-01-26 04:46 EDT-------
IBM will provide a fix for this LP

------- Comment From <email address hidden> 2021-01-27 13:50 EDT-------
I have sent a patch to the upstream mailing list for review:
https://www.redhat.com/archives/libvir-list/2021-January/msg01149.html

Adjusting affected package to qemu instead of kernel.

------- Comment From <email address hidden> 2021-01-29 07:34 EDT-------
The patch has been accepted upstream.
Here is the upstream commit id:
bd112c9e0f qemu: Add virtio related options to vsock

Michal Privoznik provided three additional patches reworking the validation which for sure create a dependency in some way.
These are the upstream commit ids:
8a4b8996f7 conf: Move virDomainCheckVirtioOptions() into domain_validate.c
c05f00666c conf: Drop empty virDomainNetDefPostParse()
19d4e46770 conf: Improve virDomainVirtioOptionsCheckABIStability()

I'm working on libvirt 7.0 atm anyway and will - for Hirsute - make this part of it.
Once complete we can think of the SRUs.

P.S. This was filed against qemu which is wrong, as those are libvirt commits.

The first try for that would be to only adapt the new code in virDomainVsockDefCheckABIStability to use the "old style" call to virDomainVirtioOptionsCheckABIStability (19d4e46770).
While that seems easy the bigger dependency is "virDomainCheckVirtioOptions" which without Michals series isn't available in src/conf/domain_validate.c
And that in turn depends on c05f00666c for PostParse.

For the 7.0 into Hirsute that isn't too much of a problem, but for the backports I'm slightly concerned. I didn't see an immediate problem - but it will have to be double checked as one would want to avoid regressions of the "yes I know it is wrong, but it used to work until your update" kind.

But one step at a time, for Hirsute I've prepared a test build as part of the 7.0 builds at:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4408/+packages

Once (if) libvirt_7.0.0-1ubuntu1~ppa7 (or later) is built there I'd appreciate if IBM could do a check of that build on an secure execution enabled machine.

------- Comment From <email address hidden> 2021-02-02 04:43 EDT-------
(In reply to comment #15)
> The first try for that would be to only adapt the new code in
> virDomainVsockDefCheckABIStability to use the "old style" call to
> virDomainVirtioOptionsCheckABIStability (19d4e46770).
> While that seems easy the bigger dependency is "virDomainCheckVirtioOptions"
> which without Michals series isn't available in src/conf/domain_validate.c
> And that in turn depends on c05f00666c for PostParse.
>
> For the 7.0 into Hirsute that isn't too much of a problem, but for the
> backports I'm slightly concerned. I didn't see an immediate problem - but it
> will have to be double checked as one would want to avoid regressions of the
> "yes I know it is wrong, but it used to work until your update" kind.
>
> But one step at a time, for Hirsute I've prepared a test build as part of
> the 7.0 builds at:
> https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4408/+packages
>
> Once (if) libvirt_7.0.0-1ubuntu1~ppa7 (or later) is built there I'd
> appreciate if IBM could do a check of that build on an secure execution
> enabled machine.

Christian,
the version I sent to the mailing list should work without Michals patches.
https://www.redhat.com/archives/libvir-list/2021-January/msg01149.html

Ok, thanks Boris - I'll consider those for the backports to 6.0 and 6.6 then.

From my POV things are not totally falling apart in Hirsute, but as mentioned I can't test the particular feature lacking secure execution.

Did anyone have the time to check the PPA I referenced above before I do an upload?

------- Comment From <email address hidden> 2021-02-03 04:41 EDT-------
> Did anyone have the time to check the PPA I referenced above before I do an
> upload?

the libvirt from ppa train 4408 seems to work fine. regarding vhost and iommu

------- Comment From <email address hidden> 2021-02-03 05:35 EDT-------
(In reply to comment #18)
> > Did anyone have the time to check the PPA I referenced above before I do an
> > upload?
>
> the libvirt from ppa train 4408 seems to work fine. regarding vhost and iommu

Just talked to Christian. He tested on a none SE system and verified the cmd line including starting a guest successfully.
Peters system is currently at 20.04 and he needs his system to stay at 20.04. He would be able able to test a 20.04 version on SE.
Do you require more tests on a SE LPAR for 21.04(hirsute)?

> Just talked to Christian. He tested on a none SE system and verified the cmd line including starting a guest successfully.
> Peters system is currently at 20.04 and he needs his system to stay at 20.04. He would be able able to test a 20.04 version on SE.
> Do you require more tests on a SE LPAR for 21.04(hirsute)?

Not strictly but I'd want to avoid verifying it properly on the F&G
SRUs to then have an upgrade regression as it is failing in Hirsute.
But If Christian (thanks) has checked the generated cmdline, that is
what we need to be somewhat sure.
Also Hirsute is the release closest to what you brought upstream, so
the chance of unexpected chaos is lower than it will be on e.g. the
backports.
So I'm ok for now, but 7.0 will take a few days more to be fully ready
- so if anyone ends up having a SE system available that could run
Hirsute give it a shot.

P.S. Being curious - can't you just "boot another disk with 21.04 into
an SE environment" - is SE so restrictive that this old nice way of
testing multiple things with a single lpar no longer works?

------- Comment From <email address hidden> 2021-02-03 07:49 EDT-------
(In reply to comment #20)
> > Just talked to Christian. He tested on a none SE system and verified the cmd line including starting a guest successfully.
> > Peters system is currently at 20.04 and he needs his system to stay at 20.04. He would be able able to test a 20.04 version on SE.
> > Do you require more tests on a SE LPAR for 21.04(hirsute)?
>
> Not strictly but I'd want to avoid verifying it properly on the F&G
> SRUs to then have an upgrade regression as it is failing in Hirsute.
> But If Christian (thanks) has checked the generated cmdline, that is
> what we need to be somewhat sure.
> Also Hirsute is the release closest to what you brought upstream, so
> the chance of unexpected chaos is lower than it will be on e.g. the
> backports.
> So I'm ok for now, but 7.0 will take a few days more to be fully ready
> - so if anyone ends up having a SE system available that could run
> Hirsute give it a shot.
>
> P.S. Being curious - can't you just "boot another disk with 21.04 into
> an SE environment" - is SE so restrictive that this old nice way of
> testing multiple things with a single lpar no longer works?

Christian was able to reran the tests successfully on a SE enabled LPAR after upgrading to 21.04.

Awesome, it also just passed regression tests again (with that change applied) on s390x.

This bug was fixed in the package libvirt - 7.0.0-1ubuntu2

---------------
libvirt (7.0.0-1ubuntu2) hirsute; urgency=medium

  * d/control: extend demotion of libvirt-lxc related dependencies to
    libvirt-login-shell

 -- Christian Ehrhardt <email address hidden> Thu, 04 Feb 2021 13:44:49 +0100

------- Comment From <email address hidden> 2021-02-08 08:15 EDT-------
@Canonical. Please add Ubuntu 20.10 and 20.04 also to this ticket.
It's required with 20.04 !

<email address hidden> 2021-02-02 04:43 EDT:
> Christian,
> the version I sent to the mailing list should work without Michals patches.
> https://www.redhat.com/archives/libvir-list/2021-January/msg01149.html

Hi,
this also still is very very noisy vs e.g. the libvirt 6.0 that is in Focal.
E.g. src/qemu/qemu_validate.c doesn't exist yet since 906fddf73 is in >v6.3.0.
I could backport it just looking at the code, but you have the experience as the author and a machine that can actually execute this. Hence I'd ask you to provide suggested backports (I'll do the packaging, just a git commit with the change that you envision) on top of those two branches:
- F: https://code.launchpad.net/~usd-import-team/ubuntu/+source/libvirt/+git/libvirt/+ref/applied/ubuntu/focal-devel
- G: https://code.launchpad.net/~usd-import-team/ubuntu/+source/libvirt/+git/libvirt/+ref/applied/ubuntu/groovy-devel

Those not only are on the versions the Ubuntu releases are on, but also have all the Ubuntu Delta that usually is applied in packaging applied.
If you want to provide a patch here or point me to a commit/branch somewhere is up to you.

------- Comment From <email address hidden> 2021-02-09 09:36 EDT-------
(In reply to comment #33)
> <email address hidden> 2021-02-02 04:43 EDT:
> > Christian,
> > the version I sent to the mailing list should work without Michals patches.
> > https://www.redhat.com/archives/libvir-list/2021-January/msg01149.html
>
> Hi,
> this also still is very very noisy vs e.g. the libvirt 6.0 that is in Focal.
> E.g. src/qemu/qemu_validate.c doesn't exist yet since 906fddf73 is in
> >v6.3.0.
> I could backport it just looking at the code, but you have the experience as
> the author and a machine that can actually execute this. Hence I'd ask you
> to provide suggested backports (I'll do the packaging, just a git commit
> with the change that you envision) on top of those two branches:
> - F:
> https://code.launchpad.net/~usd-import-team/ubuntu/+source/libvirt/+git/
> libvirt/+ref/applied/ubuntu/focal-devel
> - G:
> https://code.launchpad.net/~usd-import-team/ubuntu/+source/libvirt/+git/
> libvirt/+ref/applied/ubuntu/groovy-devel
>
> Those not only are on the versions the Ubuntu releases are on, but also have
> all the Ubuntu Delta that usually is applied in packaging applied.
> If you want to provide a patch here or point me to a commit/branch somewhere
> is up to you.

I have attached patches for focal and groovy.
The two patches have been successfully build and passed the mocked libvirt checks. Libvirts syntax-check seems pretty much as broken as before. :D

------- Comment on attachment From <email address hidden> 2021-02-09 09:31 EDT-------

My best try with focal in mind :-D

------- Comment on attachment From <email address hidden> 2021-02-09 09:32 EDT-------

My groovy approach to the patch... :-)

Thanks @Boris, taking a look ...

Thanks those backports seem to work fine.
The PPA [1] has preliminary builds for Focal and Groovy now that you can try.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4408

------- Comment From <email address hidden> 2021-02-09 10:39 EDT-------
(In reply to comment #38)
> Thanks those backports seem to work fine.
> The PPA [1] has preliminary builds for Focal and Groovy now that you can try.
>
> [1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4408

You can actually try out on a none-SE LPAR if the qemu command line parameter "iommu_platform=on" gets generated correctly.
That done we would only need additional testing in a SE LPAR if you are not sure if focal or groovy are SE ready with this patch. Please let me know.

------- Comment From <email address hidden> 2021-02-09 11:54 EDT-------
I tested libvirt 6.0.0-0ubuntu8.7~ppa1 on a z15 SE enabled LPAR in Ubuntu 20.04.
I can now add "<driver iommu='on'/>" to the vsock section in the Libvirt XML file.
Qemu then runs with the 'iommu_platform=on' argument of the vsock device driver.
Communication between the hypervisor and a SE enabled KVM guest via Vsock now works as expected.
Thanks for the quick fix!

I added the SRU description to the bug, MP review is complete and also the sniff tests on the PPA are. Thanks everyone for the participation - uploaded to F/G -unapproved for review by the SRU Team.

An upload of libvirt to groovy-proposed has been rejected from the upload queue for the following reason: "Please re-upload after building -v to include the previous test-improvement in the .changes file (it's block-proposed).".

An upload of libvirt to focal-proposed has been rejected from the upload queue for the following reason: "Please re-upload after building with -v... to include the previous test-improvement in the .changes file (it's block-proposed).".

I have yesterday uploaded a .changes including both versions and pinged sil2100.
Seems he wasn't around anymore :-/
@SRU - Please re-eval this and let me know if anything (else) is missing.

Hello bugproxy, or anyone else affected,

Accepted libvirt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.0.0-0ubuntu8.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello bugproxy, or anyone else affected,

Accepted libvirt into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.6.0-1ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Non-fixed builds do not accept the required configuration at all.

error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng
Extra element devices in interleave
Element domain failed to validate content

Failed. Try again? [y,n,i,f,?]:

But specifying cid/iommu is reuqired to be usable in SE environments.

Now I upgraded to the new builds that are in proposed.
  Unpacking libvirt-daemon-system (6.0.0-0ubuntu8.7) over (6.0.0-0ubuntu8.5) ...
  Unpacking libvirt-daemon-system (6.6.0-1ubuntu3.3) over (6.6.0-1ubuntu3.1) ...
Both worked without issues

With those the new config could be added.
That is one check passed \o/

But the second check I need IBM for.
We have discussed/thought that we can check the commandline that is generated and ignore that this is eventually to help an SE environment. But TBH in my case there is no iommu rendered in the commandline.

It is not even adding any -device vhost-vsock-ccw or such. Maybe in my restricted environment another pre-check fails.
I've had a focal LPAR around that I was able to use, there things work as expected.

ubuntu@s1lp5:~$ sudo grep -e iommu -e vsock /var/log/libvirt/qemu/h-test.log
-device vhost-vsock-ccw,id=vsock0,guest-cid=3,vhostfd=27,iommu_platform=on,devno=fe.0.0005 \

I just can't bump this to groovy atm.
So I've used a 2nd level KVM for groovy.
Finally there I can confirm as well:
ubuntu@g-test:~$ sudo grep -e iommu -e vsock /var/log/libvirt/qemu/testvm.log
-device vhost-vsock-ccw,id=vsock0,guest-cid=3,vhostfd=25,iommu_platform=on,devno=fe.0.0005 \

So - both releases verified

All autopkgtests for the newly accepted libvirt (6.6.0-1ubuntu3.3) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

libvirt-dbus/1.4.0-1ubuntu1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#libvirt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

FYI - reported test issue is resolved by now

The verification of the Stable Release Update for libvirt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package libvirt - 6.6.0-1ubuntu3.3

---------------
libvirt (6.6.0-1ubuntu3.3) groovy; urgency=medium

  * d/p/u/lp-1913266-qemu-Add-virtio-related-options-to-vsock.patch: allow
    vsock to work in secure execution environments. (LP: #1913266)

libvirt (6.6.0-1ubuntu3.2) groovy; urgency=medium

  * Improve flaky smoke-lxc test (LP: #1899180)
    - d/t/control, d/t/smoke-lxc: retry service restart and skip test if
      failing; This was flaky on some release/architectures
    - d/t/smoke-lxc: retry check_domain being flaky on arm64

 -- Christian Ehrhardt <email address hidden> Tue, 09 Feb 2021 16:14:26 +0100

This bug was fixed in the package libvirt - 6.0.0-0ubuntu8.7

---------------
libvirt (6.0.0-0ubuntu8.7) focal; urgency=medium

  * d/p/u/lp-1913266-qemu-Add-virtio-related-options-to-vsock.patch: allow
    vsock to work in secure execution environments. (LP: #1913266)

libvirt (6.0.0-0ubuntu8.6) focal; urgency=medium

  * Improve flaky smoke-lxc test (LP: #1899180)
    - d/t/control, d/t/smoke-lxc: retry service restart and skip test if
      failing; This was flaky on some release/architectures
    - d/t/smoke-lxc: retry check_domain being flaky on arm64

 -- Christian Ehrhardt <email address hidden> Tue, 09 Feb 2021 16:09:39 +0100

------- Comment From <email address hidden> 2021-03-01 06:23 EDT-------
IBM Bugzilla status->closed. Fix Released with all requested distros.