OpenStack Compute (nova)

make nova floating-ip-delete atomic with neutron

Bug #1356157 reported by Aaron Rosen on 2014-08-13

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
OpenStack Compute (nova)	Fix Released	High	Aaron Rosen	OpenStack Compute (nova) 2014.2 "juno"
Havana	Fix Released	High	Aaron Rosen	OpenStack Compute (nova) 2013.2.4
Icehouse	Fix Released	High	Aaron Rosen	OpenStack Compute (nova) 2014.1.3

Bug Description

The infra guys were noticing an issue where they were leaking floating ip
addresses. One of the reasons this would occur for them is they called
nova floating-ip-delete which first disassocates the floating-ip in neutron
and then deletes it. Because it makes two calls to neutron if the first one
succeeds and the second fails it results in the instance no longer being
associated with the floatingip. They have retry logic but they base it on
the instance and when they go to retry cleaning up the instance the floatingip
is no longer on the instance so they never delete it.

This patch fixes this issue by directly calling delete_floating_ip instead
of releasing first if using neutron as neutron allows this. I looked into
doing the same thing for nova-network but the code is written to prevent this.
This allows the operation to be atomic. I know this is sorta hackish that
we're doing this in the api layer but we do this in a few other places
too fwiw.

Tags:

Aaron Rosen (arosen) on 2014-08-13

Changed in nova:
importance:	Undecided → High
assignee:	nobody → Aaron Rosen (arosen)
tags:	added: network

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-13: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/113754

Changed in nova:
status:	New → In Progress

Matt Riedemann (mriedem) on 2014-08-13

tags:

added: icehouse-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-13: Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/114026

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-13: Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/114029

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-13: Change abandoned on nova (master)

Change abandoned by Aaron Rosen (<email address hidden>) on branch: master
Review: https://review.openstack.org/114026
Reason: squashed

OpenStack Infra (hudson-openstack) on 2014-08-27

Changed in nova:
assignee:	Aaron Rosen (arosen) → Matt Riedemann (mriedem)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-03: Change abandoned on nova (stable/icehouse)

Change abandoned by Aaron Rosen (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/114029

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-05: Fix merged to nova (master)

Reviewed: https://review.openstack.org/113754
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5f391087d1d75dd4e7532252c312256e9c3ea2b9
Submitter: Jenkins
Branch: master

commit 5f391087d1d75dd4e7532252c312256e9c3ea2b9
Author: Aaron Rosen <email address hidden>
Date: Tue Aug 12 19:02:10 2014 -0700

Make floatingip-ip-delete atomic with neutron

    The infra guys were noticing an issue where they were leaking floating ip
    addresses. One of the reasons this would occur for them is they called
    nova floating-ip-delete which first disassocates the floating-ip in neutron
    and then deletes it. Because nova-api makes two calls to neutron if the
    first one succeeds and the second fails it results in the instance no longer
    being associated with the floatingip. They have retry logic but they base it on
    the instance and when they go to retry cleaning up the instance the floatingip
    is no longer on the instance so they never delete it.

    This patch fixes this issue by creating a new method
    disassocate_and_delete_floating_ip which contains the different logic
    needed to atomically do this for neutron rather than putting the logic
    in the api layer to do something differently if one is using neutron.

This patch also adds additional code coverage for nova-network in the
delete floatingip case.

Closes-bug: 1356157
Change-Id: I53b0c9d949404288e8687b304361a74b53d69ef3

Changed in nova:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-06: Fix merged to nova (stable/icehouse)

Reviewed: https://review.openstack.org/114029
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4bc680f2be3dec8372b85f8874fd73c2b4f1b794
Submitter: Jenkins
Branch: stable/icehouse

commit 4bc680f2be3dec8372b85f8874fd73c2b4f1b794
Author: Aaron Rosen <email address hidden>
Date: Tue Aug 12 19:02:10 2014 -0700

Make floatingip-ip-delete atomic with neutron

    The infra guys were noticing an issue where they were leaking floating ip
    addresses. One of the reasons this would occur for them is they called
    nova floating-ip-delete which first disassocates the floating-ip in neutron
    and then deletes it. Because nova-api makes two calls to neutron if the
    first one succeeds and the second fails it results in the instance no longer
    being associated with the floatingip. They have retry logic but they base it on
    the instance and when they go to retry cleaning up the instance the floatingip
    is no longer on the instance so they never delete it.

This patch also adds additional code coverage for nova-network in the
delete floatingip case.

Closes-bug: 1356157

    Conflicts:
     nova/api/openstack/compute/contrib/floating_ips.py
     nova/network/base_api.py

Change-Id: I53b0c9d949404288e8687b304361a74b53d69ef3
(cherry picked from commit 5f391087d1d75dd4e7532252c312256e9c3ea2b9)

tags:

added: in-stable-icehouse

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-17: Fix proposed to nova (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/122060

Alan Pevec (apevec) on 2014-09-18

Changed in nova:
assignee:	Matt Riedemann (mriedem) → Aaron Rosen (arosen)
tags:	removed: icehouse-backport-potential in-stable-icehouse

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-19: Fix merged to nova (stable/havana)

Reviewed: https://review.openstack.org/122060
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=97028309ff7a13ea5e9b8cc353d56ec9678caaff
Submitter: Jenkins
Branch: stable/havana

commit 97028309ff7a13ea5e9b8cc353d56ec9678caaff
Author: Aaron Rosen <email address hidden>
Date: Tue Aug 12 19:02:10 2014 -0700

Make floatingip-ip-delete atomic with neutron

    The infra guys were noticing an issue where they were leaking floating ip
    addresses. One of the reasons this would occur for them is they called
    nova floating-ip-delete which first disassocates the floating-ip in neutron
    and then deletes it. Because nova-api makes two calls to neutron if the
    first one succeeds and the second fails it results in the instance no longer
    being associated with the floatingip. They have retry logic but they base it on
    the instance and when they go to retry cleaning up the instance the floatingip
    is no longer on the instance so they never delete it.

This patch also adds additional code coverage for nova-network in the
delete floatingip case.

Closes-bug: 1356157

    Conflicts:
     nova/api/openstack/compute/contrib/floating_ips.py
     nova/network/base_api.py

Change-Id: I53b0c9d949404288e8687b304361a74b53d69ef3
(cherry picked from commit 5f391087d1d75dd4e7532252c312256e9c3ea2b9)

Thierry Carrez (ttx) on 2014-10-01

Changed in nova:
milestone:	none → juno-rc1
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-10-16

Changed in nova:
milestone:	juno-rc1 → 2014.2

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.