Performing attach/detach interface on a VM sometimes results in an interface that can't be detached from the VM.
I could triage it to the corrupted instance cache info due to non-atomic update of that information.
Details on how to reproduce the bug are as follows. Since this is due to a race condition, the test can take quite a bit of time before it hits the bug.
Steps to reproduce:
1) Devstack with trunk with the following local.conf:
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service q-metering
RECLONE=yes
# and other options as set in the trunk's local
2) Create few networks:
$> neutron net-create testnet1
$> neutron net-create testnet2
$> neutron net-create testnet3
$> neutron subnet-create testnet1 192.168.1.0/24
$> neutron subnet-create testnet2 192.168.2.0/24
$> neutron subnet-create testnet3 192.168.3.0/24
2) Create a testvm in testnet1:
$> nova boot --flavor m1.tiny --image cirros-0.3.2-x86_64-uec --nic net-id=`neutron net-list | grep testnet1 | cut -f 2 -d ' '` testvm
3) Run the following shell script to attach and detach interfaces for this vm in the remaining two networks in a loop until we run into the issue at hand:
--------
#! /bin/bash
c=10000
netid1=`neutron net-list | grep testnet2 | cut -f 2 -d ' '`
netid2=`neutron net-list | grep testnet3 | cut -f 2 -d ' '`
while [ $c -gt 0 ]
do
echo "Round: " $c
echo -n "Attaching two interfaces... "
nova interface-attach --net-id $netid1 testvm
nova interface-attach --net-id $netid2 testvm
echo "Done"
echo "Sleeping until both those show up in interfaces"
waittime=0
while [ $waittime -lt 60 ]
do
count=`nova interface-list testvm | wc -l`
if [ $count -eq 7 ]
then
break
fi
sleep 2
(( waittime+=2 ))
done
echo "Waited for " $waittime " seconds"
echo "Detaching both... "
nova interface-list testvm | grep $netid1 | awk '{print "deleting ",$4; system("nova interface-detach testvm "$4 " ; sleep 2");}'
nova interface-list testvm | grep $netid2 | awk '{print "deleting ",$4; system("nova interface-detach testvm "$4 " ; sleep 2");}'
echo "Done; check interfaces are gone in a minute."
waittime=0
while [ $waittime -lt 60 ]
do
count=`nova interface-list testvm | wc -l`
echo "line count: " $count
if [ $count -eq 5 ]
then
break
fi
sleep 2
(( waittime+=2 ))
done
if [ $waittime -ge 60 ]
then
echo "bad case"
exit 1
fi
echo "Interfaces are gone"
(( c-- ))
done
---------
Eventually the test will stop with a failure ("bad case") and the interface remaining either from testnet2 or testnet3 can not be detached at all.
I could triage it to non-atomic instance info cache update. It seems heal_instance_
task runs in parallel with these API calls and due to race condition between these two, the info
cache is getting corrupted.
I put in a debug statement to print stack track for each cache update and in the following failed case you could see that the "_heal_
------
aviuser@
2014-06-03 17:13:27.870 DEBUG nova.network.
2014-06-03 17:13:27.871 DEBUG nova.network.