Live migration fails. XML error: CPU feature `wdt' specified more than once

I've hit a similar issue before. We should coordinate with the libvirt team to try to get this solved on their end, but we should also have a check on our end for older libvirt versions.

I think this problem should be resolved if we will backport the following patch to Havana:
https://review.openstack.org/#/c/65360/

I'm not sure that this is a bug with libvirt.

This bug appears to come from `get_host_capabilities` in nova/virt/libvirt/driver.py

We `getCapabilities` from the libvirt connection, then parse this into our object.
Next we call libvirt's `baselineCPU` with a XML representation of our CPU.
This returns an XML string that includes all the CPU features, including the ones we passed to it. This, to me, sounds right according to libvirt documentation.

From: http://libvirt.org/html/libvirt-libvirt.html#virConnectBaselineCPU
"Computes the most feature-rich CPU which is compatible with all given host CPUs.
If @flags includes VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES then libvirt will explicitly list all CPU features that are part of the host CPU, without this flag features that are part of the CPU model will not be listed."

We then parse all the features that are returned back into our object creating an XML document with duplicates.

Then, we later use this XML in the `_compare_cpu` function in driver.py, in which libvirt returns an error because it has received the XML from Nova with a duplicate.

In contrast to the related bug, it appears that it was caused by an error in libvirt where the baselineCPU function would error before returning the XML to us.

If I'm not misinterpreting this issue I think we should ensure we don't place duplicate CPU features into our array.

Something like the attached patch would prevent duplicates from entering our list.

Patching All path above but the problem still there. This is my nova-compute log
2014-04-19 01:41:22.842 4397 INFO nova.virt.libvirt.driver [req-1ee12afe-c0ec-4d82-a6a5-3280407ca8e1 67ac350f0a164433ac342b4960300341 a3ace7f03db4449db2b57f95fadc94ea] Instance launched has CPU info:
{"vendor": "Intel", "model": "core2duo", "arch": "x86_64", "features": ["lahf_lm", "rdtscp", "lahf_lm", "rdtscp", "lm", "nx", "syscall", "ssse3", "monitor", "pni", "sse2", "sse", "fxsr", "mmx", "clflush", "pse36", "pat", "cmov", "mca", "pge", "mtrr", "sep", "apic", "cx8", "mce", "pae", "msr", "tsc", "pse", "de", "vme", "fpu"], "topology": {"cores": 1, "threads": 1, "sockets": 1}}
2014-04-19 01:41:22.846 4397 ERROR nova.virt.libvirt.driver [req-1ee12afe-c0ec-4d82-a6a5-3280407ca8e1 67ac350f0a164433ac342b4960300341 a3ace7f03db4449db2b57f95fadc94ea] CPU doesn't have compatibility.

XML error: CPU feature `lahf_lm' specified more than once

Refer to http://libvirt.org/html/libvirt-libvirt.html#virCPUCompareResult
2014-04-19 01:41:22.868 4397 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: XML error: CPU feature `lahf_lm' specified more than once
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher payload)
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 309, in decorated_function
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-04-19 01:41:22.868 4397 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dis...

Hi Jon,

I can't duplicate this issue with my patch applied to the systems I have, it is successfully filtering out the duplicates on all 6 of my test systems.

I've attached a small bit of code to test the point where I applied the patch. This replicates a small section of the code from nova where it generates the list and causes the duplication I saw.

If this returns 1 and tells you it found a duplicate you'll need to verify the patch applied properly, then do some debugging. The patch was generated against 2014.1.1. For use in the cluster it will require all systems with nova-compute running to be patched.

If that returns clean but you still can't perform a live-migration you'll need dig a bit to figure out what's going on. The first patch by Maximiliano should ensure that it filters the duplicate XML features no mater where they are generated.

Have you tried using https://review.openstack.org/#/c/65360/ ?
This should resolve the problem.

https://review.openstack.org/#/c/65360/ does not resolve this issue - The problem persists in Nova version 2014.1-0ubuntu1, which already includes this change.

I can confirm that "ensure_added_feature_is_unique.patch" does fix the problem.

Fix proposed to branch: master
Review: https://review.openstack.org/97782

Reviewed: https://review.openstack.org/97782
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=60c899b9da8d64b4a5979b69c43c77ed9d5bf248
Submitter: Jenkins
Branch: master

commit 60c899b9da8d64b4a5979b69c43c77ed9d5bf248
Author: Vladik Romanovsky <email address hidden>
Date: Tue Jun 3 14:58:06 2014 -0400

    libvirt: convert cpu features attribute from list to a set

    Currently, the cpu features list which is being sent to libvirt,
    when creating a domain or calling compareCPU, must contain only
    unique entries. Multiple issues arise when we are updating the
    features attribute in LibvirtConfigCPU class (for example during
    migration).

    This change will change the features attribute from being a list
    to a set. This make the LibvirtConfigCPU class keep only unique
    features.
    Adjusting the LibvirtConfigCPUFeature class to support set
    operations by overriding the __eq__, __ne__ and __hash__
    methods.

    Closes-Bug: #1303536
    Change-Id: I6350fe0e827c860aea77cc4fe56f18f5c1483580

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/100632

Reviewed: https://review.openstack.org/100632
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6b7cb1ae972da835cfdeaca6f3d5556cffe89ff3
Submitter: Jenkins
Branch: stable/icehouse

commit 6b7cb1ae972da835cfdeaca6f3d5556cffe89ff3
Author: Vladik Romanovsky <email address hidden>
Date: Tue Jun 3 14:58:06 2014 -0400

    libvirt: convert cpu features attribute from list to a set

    Currently, the cpu features list which is being sent to libvirt,
    when creating a domain or calling compareCPU, must contain only
    unique entries. Multiple issues arise when we are updating the
    features attribute in LibvirtConfigCPU class (for example during
    migration).

    This change will change the features attribute from being a list
    to a set. This make the LibvirtConfigCPU class keep only unique
    features.
    Adjusting the LibvirtConfigCPUFeature class to support set
    operations by overriding the __eq__, __ne__ and __hash__
    methods.

    Closes-Bug: #1303536
    Change-Id: I6350fe0e827c860aea77cc4fe56f18f5c1483580
    Cherry-pick-x: 60c899b9da8d64b4a5979b69c43c77ed9d5bf248