VMs fail to start when Ceph is used as a backend for ephemeral drives

Bug #1362221 reported by Roman Podoliaka
20
This bug affects 6 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Roman Podoliaka
Icehouse
Fix Released
High
Yaguang Tang

Bug Description

VMs' drives placement in Ceph option has been chosen (libvirt.images_types == 'rbd').

When user creates a flavor and specifies:
   - root drive size >0
   - ephemeral drive size >0 (important)

and tries to boot a VM, he gets "no valid host was found" in the scheduler log:

Error from last host: node-3.int.host.com (node node-3.int.host.com): [u'Traceback (most recent call last):\n', u'
 File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1305, in _build_instance\n set_access_ip=set_access_ip)\n', u' File "/usr/l
ib/python2.6/site-packages/nova/compute/manager.py", line 393, in decorated_function\n return function(self, context, *args, **kwargs)\n', u' File
 "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1717, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instanc
e)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, se
lf.tb)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1714, in _spawn\n block_device_info)\n', u' File "/usr/lib/py
thon2.6/site-packages/nova/virt/libvirt/driver.py", line 2259, in spawn\n admin_pass=admin_password)\n', u' File "/usr/lib/python2.6/site-packages
/nova/virt/libvirt/driver.py", line 2648, in _create_image\n ephemeral_size=ephemeral_gb)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/
libvirt/imagebackend.py", line 186, in cache\n *args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py",
line 587, in create_image\n prepare_template(target=base, max_size=size, *args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/opens
tack/common/lockutils.py", line 249, in inner\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebac
kend.py", line 176, in fetch_func_sync\n fetch_func(target=target, *args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvir
t/driver.py", line 2458, in _create_ephemeral\n disk.mkfs(os_type, fs_label, target, run_as_root=is_block_dev)\n', u' File "/usr/lib/python2.6/sit
e-packages/nova/virt/disk/api.py", line 117, in mkfs\n utils.mkfs(default_fs, target, fs_label, run_as_root=run_as_root)\n', u' File "/usr/lib/pyt
hon2.6/site-packages/nova/utils.py", line 856, in mkfs\n execute(*args, run_as_root=run_as_root)\n', u' File "/usr/lib/python2.6/site-packages/nov
a/utils.py", line 165, in execute\n return processutils.execute(*cmd, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/commo
n/processutils.py", line 193, in execute\n cmd=\' \'.join(cmd))\n', u"ProcessExecutionError: Unexpected error while running command.\nCommand: sudo
 nova-rootwrap /etc/nova/rootwrap.conf mkfs -t ext3 -F -L ephemeral0 /var/lib/nova/instances/_base/ephemeral_1_default\nExit code: 1\nStdout: ''\nStde
rr: 'mke2fs 1.41.12 (17-May-2010)\\nmkfs.ext3: No such file or directory while trying to determine filesystem size\\n'\n"]

Changed in nova:
assignee: nobody → Roman Podoliaka (rpodolyaka)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/117282

Changed in nova:
status: New → In Progress
Yaguang Tang (heut2008)
Changed in nova:
importance: Undecided → High
tags: added: icehouse-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/117282
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e9eb1d69bf75c22b0ad4b50e4892a5644c78cf68
Submitter: Jenkins
Branch: master

commit e9eb1d69bf75c22b0ad4b50e4892a5644c78cf68
Author: Roman Podoliaka <email address hidden>
Date: Fri Aug 22 20:01:33 2014 +0300

    Fix instance boot when Ceph is used for ephemeral storage

    is_block_dev attribute was mistakenly set to True for RBD images
    (we don't actually map RBD images to block devices in the *host*
    system, but only in *guests*, so as far as the host system is
    concerned an RBD image must not be treated like a system block
    device). This led to a situation when mkfs was called for a non
    existing file and failed when trying to create an ephemeral fs.

    Closes-Bug: #1362221

    Change-Id: I54c0d117a53bb61f278b2e137bd29a595548f5a1

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/119520

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/icehouse)

Reviewed: https://review.openstack.org/119520
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1613cd99345e51e314bf011f47654ed730138f64
Submitter: Jenkins
Branch: stable/icehouse

commit 1613cd99345e51e314bf011f47654ed730138f64
Author: Roman Podoliaka <email address hidden>
Date: Fri Aug 22 20:01:33 2014 +0300

    Fix instance boot when Ceph is used for ephemeral storage

    is_block_dev attribute was mistakenly set to True for RBD images
    (we don't actually map RBD images to block devices in the *host*
    system, but only in *guests*, so as far as the host system is
    concerned an RBD image must not be treated like a system block
    device). This led to a situation when mkfs was called for a non
    existing file and failed when trying to create an ephemeral fs.

    Closes-Bug: #1362221

    Change-Id: I54c0d117a53bb61f278b2e137bd29a595548f5a1
    (cherry picked from commit e9eb1d69bf75c22b0ad4b50e4892a5644c78cf68)

tags: added: in-stable-icehouse
Thierry Carrez (ttx)
Changed in nova:
milestone: none → juno-rc1
status: Fix Committed → Fix Released
Ante Karamatić (ivoks)
tags: added: cts
Yaguang Tang (heut2008)
tags: removed: icehouse-backport-potential
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-rc1 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.