cloud-init

Metadata lazy loaded (even when disabled)

Bug #1068801 reported by Joshua Harlow
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
Medium
Unassigned

Bug Description

A situation I was thinking probably is a bug.

Looking at the boto fetching of metadata, they lazily load the dictionary that forms the instances metadata (and subsequent contained 'sub' dictionaries as well). When the ec2 datasource is serialized it only contains whatever was loaded up to that point (depending on which fields were accessed). Now say the ec2 iptables rule is applied, further lazy loadings accessing contained sub entries of the metadata will fail loading. This would be counterintuitive to modules that expect fields to exist (and aren't prepared for lazy loading that happens in the lazy loading wrapper).

Perhaps before serialization or in our custom ec2_utils (tbd) the full metadata could be traversed and stored so that this situation does not happen.

Related branches

lp:~harlowja/cloud-init/url-ssl-fixings
Scott Moser: Needs Fixing
Diff: 890 lines (+316/-163)
15 files modified
Requires (+6/-5)
cloudinit/config/cc_phone_home.py (+3/-2)
cloudinit/config/cc_power_state_change.py (+1/-1)
cloudinit/distros/parsers/resolv_conf.py (+2/-2)
cloudinit/ec2_utils.py (+5/-0)
cloudinit/sources/DataSourceCloudStack.py (+6/-5)
cloudinit/sources/DataSourceEc2.py (+7/-6)
cloudinit/sources/DataSourceMAAS.py (+26/-19)
cloudinit/url_helper.py (+159/-98)
cloudinit/user_data.py (+3/-2)
cloudinit/util.py (+79/-10)
setup.py (+3/-2)
tests/unittests/test__init__.py (+5/-5)
tests/unittests/test_datasource/test_maas.py (+9/-6)
tests/unittests/test_util.py (+2/-0)
lp:~harlowja/cloud-init/boto-metadata-fixings
cloud-init Commiters: Pending requested
Diff: 128 lines (+76/-12)
3 files modified
cloudinit/ec2_utils.py (+66/-0)
cloudinit/sources/DataSourceCloudStack.py (+5/-6)
cloudinit/sources/DataSourceEc2.py (+5/-6)
lp:~cloud-init-dev/cloud-init/trunk
Chad Smith (community): Needs Fixing
Hannes: Pending requested
Diff: 51583 lines (+11/-49380) (has conflicts)
424 files modified
.bzrignore (+0/-4)
ChangeLog (+0/-761)
HACKING.rst (+0/-48)
LICENSE (+0/-674)
MANIFEST.in (+0/-8)
Makefile (+0/-82)
README (+11/-0)
TODO.rst (+0/-43)
bin/cloud-init (+0/-673)
cloudinit/__init__.py (+0/-21)
cloudinit/cloud.py (+0/-109)
cloudinit/config/__init__.py (+0/-58)
cloudinit/config/cc_apt_configure.py (+0/-292)
cloudinit/config/cc_apt_pipelining.py (+0/-57)
cloudinit/config/cc_bootcmd.py (+0/-54)
cloudinit/config/cc_byobu.py (+0/-80)
cloudinit/config/cc_ca_certs.py (+0/-104)
cloudinit/config/cc_chef.py (+0/-342)
cloudinit/config/cc_debug.py (+0/-109)
cloudinit/config/cc_disable_ec2_metadata.py (+0/-36)
cloudinit/config/cc_emit_upstart.py (+0/-69)
cloudinit/config/cc_fan.py (+0/-101)
cloudinit/config/cc_final_message.py (+0/-73)
cloudinit/config/cc_foo.py (+0/-52)
cloudinit/config/cc_growpart.py (+0/-300)
cloudinit/config/cc_grub_dpkg.py (+0/-73)
cloudinit/config/cc_keys_to_console.py (+0/-62)
cloudinit/config/cc_landscape.py (+0/-99)
cloudinit/config/cc_locale.py (+0/-37)
cloudinit/config/cc_lxd.py (+0/-176)
cloudinit/config/cc_mcollective.py (+0/-88)
cloudinit/config/cc_migrator.py (+0/-85)
cloudinit/config/cc_mounts.py (+0/-405)
cloudinit/config/cc_package_update_upgrade_install.py (+0/-99)
cloudinit/config/cc_phone_home.py (+0/-122)
cloudinit/config/cc_power_state_change.py (+0/-223)
cloudinit/config/cc_puppet.py (+0/-118)
cloudinit/config/cc_resizefs.py (+0/-185)
cloudinit/config/cc_resolv_conf.py (+0/-116)
cloudinit/config/cc_rh_subscription.py (+0/-408)
cloudinit/config/cc_rightscale_userdata.py (+0/-102)
cloudinit/config/cc_rsyslog.py (+0/-366)
cloudinit/config/cc_runcmd.py (+0/-38)
cloudinit/config/cc_salt_minion.py (+0/-59)
cloudinit/config/cc_scripts_per_boot.py (+0/-41)
cloudinit/config/cc_scripts_per_instance.py (+0/-41)
cloudinit/config/cc_scripts_per_once.py (+0/-41)
cloudinit/config/cc_scripts_user.py (+0/-42)
cloudinit/config/cc_scripts_vendor.py (+0/-43)
cloudinit/config/cc_seed_random.py (+0/-94)
cloudinit/config/cc_set_hostname.py (+0/-37)
cloudinit/config/cc_set_passwords.py (+0/-167)
cloudinit/config/cc_snappy.py (+0/-304)
cloudinit/config/cc_ssh.py (+0/-142)
cloudinit/config/cc_ssh_authkey_fingerprints.py (+0/-105)
cloudinit/config/cc_ssh_import_id.py (+0/-99)
cloudinit/config/cc_timezone.py (+0/-39)
cloudinit/config/cc_ubuntu_init_switch.py (+0/-162)
cloudinit/config/cc_update_etc_hosts.py (+0/-60)
cloudinit/config/cc_update_hostname.py (+0/-43)
cloudinit/config/cc_users_groups.py (+0/-34)
cloudinit/config/cc_write_files.py (+0/-103)
cloudinit/config/cc_yum_add_repo.py (+0/-107)
cloudinit/cs_utils.py (+0/-105)
cloudinit/distros/__init__.py (+0/-950)
cloudinit/distros/arch.py (+0/-201)
cloudinit/distros/debian.py (+0/-227)
cloudinit/distros/fedora.py (+0/-31)
cloudinit/distros/freebsd.py (+0/-417)
cloudinit/distros/gentoo.py (+0/-160)
cloudinit/distros/net_util.py (+0/-182)
cloudinit/distros/parsers/__init__.py (+0/-28)
cloudinit/distros/parsers/hostname.py (+0/-88)
cloudinit/distros/parsers/hosts.py (+0/-92)
cloudinit/distros/parsers/resolv_conf.py (+0/-169)
cloudinit/distros/parsers/sys_conf.py (+0/-113)
cloudinit/distros/rhel.py (+0/-222)
cloudinit/distros/rhel_util.py (+0/-89)
cloudinit/distros/sles.py (+0/-179)
cloudinit/distros/ubuntu.py (+0/-31)
cloudinit/ec2_utils.py (+0/-201)
cloudinit/filters/__init__.py (+0/-21)
cloudinit/filters/launch_index.py (+0/-75)
cloudinit/handlers/__init__.py (+0/-274)
cloudinit/handlers/boot_hook.py (+0/-70)
cloudinit/handlers/cloud_config.py (+0/-163)
cloudinit/handlers/shell_script.py (+0/-55)
cloudinit/handlers/upstart_job.py (+0/-119)
cloudinit/helpers.py (+0/-458)
cloudinit/importer.py (+0/-58)
cloudinit/log.py (+0/-155)
cloudinit/mergers/__init__.py (+0/-166)
cloudinit/mergers/m_dict.py (+0/-88)
cloudinit/mergers/m_list.py (+0/-89)
cloudinit/mergers/m_str.py (+0/-46)
cloudinit/net/__init__.py (+0/-771)
cloudinit/net/network_state.py (+0/-447)
cloudinit/net/udev.py (+0/-54)
cloudinit/netinfo.py (+0/-249)
cloudinit/patcher.py (+0/-58)
cloudinit/registry.py (+0/-37)
cloudinit/reporting/__init__.py (+0/-42)
cloudinit/reporting/events.py (+0/-248)
cloudinit/reporting/handlers.py (+0/-91)
cloudinit/safeyaml.py (+0/-32)
cloudinit/settings.py (+0/-68)
cloudinit/signal_handler.py (+0/-71)
cloudinit/sources/DataSourceAltCloud.py (+0/-293)
cloudinit/sources/DataSourceAzure.py (+0/-651)
cloudinit/sources/DataSourceBigstep.py (+0/-57)
cloudinit/sources/DataSourceCloudSigma.py (+0/-141)
cloudinit/sources/DataSourceCloudStack.py (+0/-253)
cloudinit/sources/DataSourceConfigDrive.py (+0/-424)
cloudinit/sources/DataSourceDigitalOcean.py (+0/-110)
cloudinit/sources/DataSourceEc2.py (+0/-211)
cloudinit/sources/DataSourceGCE.py (+0/-167)
cloudinit/sources/DataSourceMAAS.py (+0/-353)
cloudinit/sources/DataSourceNoCloud.py (+0/-336)
cloudinit/sources/DataSourceNone.py (+0/-57)
cloudinit/sources/DataSourceOVF.py (+0/-429)
cloudinit/sources/DataSourceOpenNebula.py (+0/-455)
cloudinit/sources/DataSourceOpenStack.py (+0/-173)
cloudinit/sources/DataSourceSmartOS.py (+0/-536)
cloudinit/sources/__init__.py (+0/-337)
cloudinit/sources/helpers/__init__.py (+0/-13)
cloudinit/sources/helpers/azure.py (+0/-279)
cloudinit/sources/helpers/openstack.py (+0/-499)
cloudinit/sources/helpers/vmware/__init__.py (+0/-13)
cloudinit/sources/helpers/vmware/imc/__init__.py (+0/-13)
cloudinit/sources/helpers/vmware/imc/boot_proto.py (+0/-25)
cloudinit/sources/helpers/vmware/imc/config.py (+0/-95)
cloudinit/sources/helpers/vmware/imc/config_file.py (+0/-129)
cloudinit/sources/helpers/vmware/imc/config_namespace.py (+0/-25)
cloudinit/sources/helpers/vmware/imc/config_nic.py (+0/-247)
cloudinit/sources/helpers/vmware/imc/config_source.py (+0/-23)
cloudinit/sources/helpers/vmware/imc/guestcust_error.py (+0/-24)
cloudinit/sources/helpers/vmware/imc/guestcust_event.py (+0/-27)
cloudinit/sources/helpers/vmware/imc/guestcust_state.py (+0/-25)
cloudinit/sources/helpers/vmware/imc/guestcust_util.py (+0/-128)
cloudinit/sources/helpers/vmware/imc/ipv4_mode.py (+0/-45)
cloudinit/sources/helpers/vmware/imc/nic.py (+0/-147)
cloudinit/sources/helpers/vmware/imc/nic_base.py (+0/-154)
cloudinit/ssh_util.py (+0/-314)
cloudinit/stages.py (+0/-844)
cloudinit/templater.py (+0/-149)
cloudinit/type_utils.py (+0/-52)
cloudinit/url_helper.py (+0/-509)
cloudinit/user_data.py (+0/-352)
cloudinit/util.py (+0/-2226)
cloudinit/version.py (+0/-27)
config/cloud.cfg (+0/-115)
config/cloud.cfg-freebsd (+0/-88)
config/cloud.cfg.d/05_logging.cfg (+0/-66)
config/cloud.cfg.d/README (+0/-3)
doc/README (+0/-4)
doc/examples/cloud-config-add-apt-repos.txt (+0/-34)
doc/examples/cloud-config-archive-launch-index.txt (+0/-30)
doc/examples/cloud-config-archive.txt (+0/-16)
doc/examples/cloud-config-boot-cmds.txt (+0/-15)
doc/examples/cloud-config-ca-certs.txt (+0/-31)
doc/examples/cloud-config-chef-oneiric.txt (+0/-90)
doc/examples/cloud-config-chef.txt (+0/-95)
doc/examples/cloud-config-datasources.txt (+0/-73)
doc/examples/cloud-config-disk-setup.txt (+0/-251)
doc/examples/cloud-config-final-message.txt (+0/-7)
doc/examples/cloud-config-gluster.txt (+0/-18)
doc/examples/cloud-config-growpart.txt (+0/-31)
doc/examples/cloud-config-install-packages.txt (+0/-15)
doc/examples/cloud-config-landscape.txt (+0/-22)
doc/examples/cloud-config-launch-index.txt (+0/-23)
doc/examples/cloud-config-lxd.txt (+0/-55)
doc/examples/cloud-config-mcollective.txt (+0/-49)
doc/examples/cloud-config-mount-points.txt (+0/-46)
doc/examples/cloud-config-phone-home.txt (+0/-14)
doc/examples/cloud-config-power-state.txt (+0/-40)
doc/examples/cloud-config-puppet.txt (+0/-51)
doc/examples/cloud-config-reporting.txt (+0/-17)
doc/examples/cloud-config-resolv-conf.txt (+0/-20)
doc/examples/cloud-config-rh_subscription.txt (+0/-49)
doc/examples/cloud-config-rsyslog.txt (+0/-46)
doc/examples/cloud-config-run-cmds.txt (+0/-22)
doc/examples/cloud-config-salt-minion.txt (+0/-53)
doc/examples/cloud-config-seed-random.txt (+0/-32)
doc/examples/cloud-config-ssh-keys.txt (+0/-46)
doc/examples/cloud-config-update-apt.txt (+0/-7)
doc/examples/cloud-config-update-packages.txt (+0/-8)
doc/examples/cloud-config-user-groups.txt (+0/-109)
doc/examples/cloud-config-vendor-data.txt (+0/-16)
doc/examples/cloud-config-write-files.txt (+0/-33)
doc/examples/cloud-config-yum-repo.txt (+0/-20)
doc/examples/cloud-config.txt (+0/-645)
doc/examples/include-once.txt (+0/-7)
doc/examples/include.txt (+0/-5)
doc/examples/kernel-cmdline.txt (+0/-18)
doc/examples/part-handler-v2.txt (+0/-38)
doc/examples/part-handler.txt (+0/-23)
doc/examples/plain-ignored.txt (+0/-2)
doc/examples/seed/README (+0/-22)
doc/examples/seed/meta-data (+0/-30)
doc/examples/seed/user-data (+0/-3)
doc/examples/upstart-cloud-config.txt (+0/-12)
doc/examples/upstart-rclocal.txt (+0/-12)
doc/examples/user-script.txt (+0/-8)
doc/merging.rst (+0/-188)
doc/rtd/conf.py (+0/-77)
doc/rtd/index.rst (+0/-31)
doc/rtd/static/logo.svg (+0/-89)
doc/rtd/topics/availability.rst (+0/-20)
doc/rtd/topics/capabilities.rst (+0/-24)
doc/rtd/topics/datasources.rst (+0/-200)
doc/rtd/topics/dir_layout.rst (+0/-81)
doc/rtd/topics/examples.rst (+0/-133)
doc/rtd/topics/format.rst (+0/-159)
doc/rtd/topics/hacking.rst (+0/-1)
doc/rtd/topics/merging.rst (+0/-5)
doc/rtd/topics/modules.rst (+0/-342)
doc/rtd/topics/moreinfo.rst (+0/-12)
doc/sources/altcloud/README.rst (+0/-87)
doc/sources/azure/README.rst (+0/-134)
doc/sources/cloudsigma/README.rst (+0/-38)
doc/sources/cloudstack/README.rst (+0/-29)
doc/sources/configdrive/README.rst (+0/-123)
doc/sources/digitalocean/README.rst (+0/-21)
doc/sources/kernel-cmdline.txt (+0/-48)
doc/sources/nocloud/README.rst (+0/-71)
doc/sources/opennebula/README.rst (+0/-142)
doc/sources/openstack/README.rst (+0/-24)
doc/sources/ovf/README (+0/-83)
doc/sources/ovf/example/ovf-env.xml (+0/-46)
doc/sources/ovf/example/ubuntu-server.ovf (+0/-130)
doc/sources/ovf/make-iso (+0/-156)
doc/sources/ovf/ovf-env.xml.tmpl (+0/-28)
doc/sources/ovf/ovfdemo.pem (+0/-27)
doc/sources/ovf/user-data (+0/-7)
doc/sources/smartos/README.rst (+0/-149)
doc/status.txt (+0/-53)
doc/userdata.txt (+0/-79)
doc/var-lib-cloud.txt (+0/-63)
doc/vendordata.txt (+0/-53)
packages/bddeb (+0/-260)
packages/brpm (+0/-277)
packages/debian/changelog.in (+0/-6)
packages/debian/cloud-init.postinst (+0/-16)
packages/debian/cloud-init.preinst (+0/-20)
packages/debian/compat (+0/-1)
packages/debian/control.in (+0/-29)
packages/debian/copyright (+0/-29)
packages/debian/dirs (+0/-6)
packages/debian/rules.in (+0/-23)
packages/debian/watch (+0/-2)
packages/redhat/cloud-init.spec.in (+0/-200)
packages/suse/cloud-init.spec.in (+0/-163)
requirements.txt (+0/-36)
setup.py (+0/-215)
systemd/cloud-config.service (+0/-16)
systemd/cloud-config.target (+0/-11)
systemd/cloud-final.service (+0/-17)
systemd/cloud-init-generator (+0/-133)
systemd/cloud-init-local.service (+0/-22)
systemd/cloud-init.service (+0/-18)
systemd/cloud-init.target (+0/-6)
sysvinit/debian/cloud-config (+0/-64)
sysvinit/debian/cloud-final (+0/-66)
sysvinit/debian/cloud-init (+0/-64)
sysvinit/debian/cloud-init-local (+0/-63)
sysvinit/freebsd/cloudconfig (+0/-35)
sysvinit/freebsd/cloudfinal (+0/-35)
sysvinit/freebsd/cloudinit (+0/-35)
sysvinit/freebsd/cloudinitlocal (+0/-35)
sysvinit/gentoo/cloud-config (+0/-13)
sysvinit/gentoo/cloud-final (+0/-11)
sysvinit/gentoo/cloud-init (+0/-12)
sysvinit/gentoo/cloud-init-local (+0/-13)
sysvinit/redhat/cloud-config (+0/-121)
sysvinit/redhat/cloud-final (+0/-121)
sysvinit/redhat/cloud-init (+0/-121)
sysvinit/redhat/cloud-init-local (+0/-124)
templates/chef_client.rb.tmpl (+0/-58)
templates/hosts.debian.tmpl (+0/-26)
templates/hosts.freebsd.tmpl (+0/-24)
templates/hosts.redhat.tmpl (+0/-24)
templates/hosts.suse.tmpl (+0/-26)
templates/resolv.conf.tmpl (+0/-30)
templates/sources.list.debian.tmpl (+0/-32)
templates/sources.list.ubuntu.tmpl (+0/-58)
test-requirements.txt (+0/-17)
tests/configs/sample1.yaml (+0/-52)
tests/data/filter_cloud_multipart.yaml (+0/-30)
tests/data/filter_cloud_multipart_1.email (+0/-11)
tests/data/filter_cloud_multipart_2.email (+0/-39)
tests/data/filter_cloud_multipart_header.email (+0/-11)
tests/data/merge_sources/expected1.yaml (+0/-1)
tests/data/merge_sources/expected10.yaml (+0/-7)
tests/data/merge_sources/expected11.yaml (+0/-5)
tests/data/merge_sources/expected12.yaml (+0/-5)
tests/data/merge_sources/expected2.yaml (+0/-3)
tests/data/merge_sources/expected3.yaml (+0/-1)
tests/data/merge_sources/expected4.yaml (+0/-2)
tests/data/merge_sources/expected5.yaml (+0/-7)
tests/data/merge_sources/expected6.yaml (+0/-9)
tests/data/merge_sources/expected7.yaml (+0/-38)
tests/data/merge_sources/expected8.yaml (+0/-7)
tests/data/merge_sources/expected9.yaml (+0/-5)
tests/data/merge_sources/source1-1.yaml (+0/-3)
tests/data/merge_sources/source1-2.yaml (+0/-5)
tests/data/merge_sources/source10-1.yaml (+0/-6)
tests/data/merge_sources/source10-2.yaml (+0/-6)
tests/data/merge_sources/source11-1.yaml (+0/-5)
tests/data/merge_sources/source11-2.yaml (+0/-3)
tests/data/merge_sources/source11-3.yaml (+0/-3)
tests/data/merge_sources/source12-1.yaml (+0/-8)
tests/data/merge_sources/source12-2.yaml (+0/-5)
tests/data/merge_sources/source2-1.yaml (+0/-6)
tests/data/merge_sources/source2-2.yaml (+0/-5)
tests/data/merge_sources/source3-1.yaml (+0/-4)
tests/data/merge_sources/source3-2.yaml (+0/-4)
tests/data/merge_sources/source4-1.yaml (+0/-3)
tests/data/merge_sources/source4-2.yaml (+0/-6)
tests/data/merge_sources/source5-1.yaml (+0/-6)
tests/data/merge_sources/source5-2.yaml (+0/-8)
tests/data/merge_sources/source6-1.yaml (+0/-5)
tests/data/merge_sources/source6-2.yaml (+0/-8)
tests/data/merge_sources/source7-1.yaml (+0/-27)
tests/data/merge_sources/source7-2.yaml (+0/-17)
tests/data/merge_sources/source8-1.yaml (+0/-7)
tests/data/merge_sources/source8-2.yaml (+0/-6)
tests/data/merge_sources/source9-1.yaml (+0/-5)
tests/data/merge_sources/source9-2.yaml (+0/-6)
tests/data/mountinfo_precise_ext4.txt (+0/-24)
tests/data/mountinfo_raring_btrfs.txt (+0/-13)
tests/data/roots/simple_ubuntu/etc/networks/interfaces (+0/-3)
tests/data/user_data.1.txt (+0/-15)
tests/data/vmware/cust-dhcp-2nic.cfg (+0/-34)
tests/data/vmware/cust-static-2nic.cfg (+0/-39)
tests/unittests/helpers.py (+0/-346)
tests/unittests/test__init__.py (+0/-220)
tests/unittests/test_builtin_handlers.py (+0/-73)
tests/unittests/test_cli.py (+0/-54)
tests/unittests/test_cs_util.py (+0/-84)
tests/unittests/test_data.py (+0/-559)
tests/unittests/test_datasource/test_altcloud.py (+0/-452)
tests/unittests/test_datasource/test_azure.py (+0/-646)
tests/unittests/test_datasource/test_azure_helper.py (+0/-421)
tests/unittests/test_datasource/test_cloudsigma.py (+0/-99)
tests/unittests/test_datasource/test_cloudstack.py (+0/-87)
tests/unittests/test_datasource/test_configdrive.py (+0/-407)
tests/unittests/test_datasource/test_digitalocean.py (+0/-127)
tests/unittests/test_datasource/test_gce.py (+0/-166)
tests/unittests/test_datasource/test_maas.py (+0/-163)
tests/unittests/test_datasource/test_nocloud.py (+0/-187)
tests/unittests/test_datasource/test_opennebula.py (+0/-300)
tests/unittests/test_datasource/test_openstack.py (+0/-347)
tests/unittests/test_datasource/test_smartos.py (+0/-585)
tests/unittests/test_distros/test_generic.py (+0/-233)
tests/unittests/test_distros/test_hostname.py (+0/-38)
tests/unittests/test_distros/test_hosts.py (+0/-41)
tests/unittests/test_distros/test_netconfig.py (+0/-321)
tests/unittests/test_distros/test_resolv.py (+0/-67)
tests/unittests/test_distros/test_sysconfig.py (+0/-82)
tests/unittests/test_distros/test_user_data_normalize.py (+0/-297)
tests/unittests/test_ec2_util.py (+0/-139)
tests/unittests/test_filters/test_launch_index.py (+0/-132)
tests/unittests/test_handler/test_handler_apt_configure.py (+0/-109)
tests/unittests/test_handler/test_handler_ca_certs.py (+0/-271)
tests/unittests/test_handler/test_handler_chef.py (+0/-192)
tests/unittests/test_handler/test_handler_debug.py (+0/-81)
tests/unittests/test_handler/test_handler_disk_setup.py (+0/-30)
tests/unittests/test_handler/test_handler_growpart.py (+0/-220)
tests/unittests/test_handler/test_handler_locale.py (+0/-67)
tests/unittests/test_handler/test_handler_lxd.py (+0/-134)
tests/unittests/test_handler/test_handler_mounts.py (+0/-133)
tests/unittests/test_handler/test_handler_power_state.py (+0/-127)
tests/unittests/test_handler/test_handler_rsyslog.py (+0/-174)
tests/unittests/test_handler/test_handler_seed_random.py (+0/-227)
tests/unittests/test_handler/test_handler_set_hostname.py (+0/-72)
tests/unittests/test_handler/test_handler_snappy.py (+0/-306)
tests/unittests/test_handler/test_handler_timezone.py (+0/-76)
tests/unittests/test_handler/test_handler_write_files.py (+0/-112)
tests/unittests/test_handler/test_handler_yum_add_repo.py (+0/-68)
tests/unittests/test_helpers.py (+0/-33)
tests/unittests/test_merging.py (+0/-257)
tests/unittests/test_net.py (+0/-130)
tests/unittests/test_pathprefix2dict.py (+0/-44)
tests/unittests/test_registry.py (+0/-28)
tests/unittests/test_reporting.py (+0/-369)
tests/unittests/test_rh_subscription.py (+0/-214)
tests/unittests/test_runs/test_merge_run.py (+0/-54)
tests/unittests/test_runs/test_simple_run.py (+0/-81)
tests/unittests/test_sshutil.py (+0/-171)
tests/unittests/test_templating.py (+0/-119)
tests/unittests/test_util.py (+0/-489)
tests/unittests/test_vmware_config_file.py (+0/-103)
tools/21-cloudinit.conf (+0/-6)
tools/Z99-cloud-locale-test.sh (+0/-98)
tools/build-on-freebsd (+0/-66)
tools/ccfg-merge-debug (+0/-90)
tools/cloud-init-per (+0/-60)
tools/hacking.py (+0/-170)
tools/make-dist-tarball (+0/-21)
tools/make-mime.py (+0/-60)
tools/make-tarball (+0/-39)
tools/mock-meta.py (+0/-454)
tools/motd-hook (+0/-35)
tools/read-dependencies (+0/-29)
tools/read-version (+0/-26)
tools/run-pep8 (+0/-22)
tools/run-pyflakes (+0/-18)
tools/run-pyflakes3 (+0/-2)
tools/tox-venv (+0/-42)
tools/uncloud-init (+0/-141)
tools/validate-yaml.py (+0/-25)
tools/write-ssh-key-fingerprints (+0/-38)
tox.ini (+0/-35)
udev/66-azure-ephemeral.rules (+0/-18)
udev/79-cloud-init-net-wait.rules (+0/-10)
udev/cloud-init-wait (+0/-70)
upstart/cloud-config.conf (+0/-9)
upstart/cloud-final.conf (+0/-10)
upstart/cloud-init-blocknet.conf (+0/-83)
upstart/cloud-init-container.conf (+0/-57)
upstart/cloud-init-local.conf (+0/-16)
upstart/cloud-init-nonet.conf (+0/-66)
upstart/cloud-init.conf (+0/-9)
upstart/cloud-log-shutdown.conf (+0/-19)
JJ Asghar: Pending requested
Diff: 0 lines
Scott Moser: Pending requested
Diff: 3061 lines (+2239/-175)
39 files modified
ChangeLog (+39/-9)
cloud-init.py (+31/-1)
cloudinit/CloudConfig/__init__.py (+1/-1)
cloudinit/CloudConfig/cc_apt_pipelining.py (+53/-0)
cloudinit/CloudConfig/cc_ca_certs.py (+3/-1)
cloudinit/CloudConfig/cc_chef.py (+5/-5)
cloudinit/CloudConfig/cc_landscape.py (+5/-0)
cloudinit/CloudConfig/cc_resizefs.py (+31/-12)
cloudinit/CloudConfig/cc_salt_minion.py (+2/-1)
cloudinit/CloudConfig/cc_update_etc_hosts.py (+1/-1)
cloudinit/DataSource.py (+4/-1)
cloudinit/DataSourceCloudStack.py (+92/-0)
cloudinit/DataSourceConfigDrive.py (+231/-0)
cloudinit/DataSourceEc2.py (+2/-84)
cloudinit/DataSourceMAAS.py (+345/-0)
cloudinit/DataSourceNoCloud.py (+75/-2)
cloudinit/DataSourceOVF.py (+1/-1)
cloudinit/SshUtil.py (+2/-0)
cloudinit/UserDataHandler.py (+2/-2)
cloudinit/__init__.py (+44/-9)
cloudinit/netinfo.py (+5/-5)
cloudinit/util.py (+253/-30)
config/cloud.cfg (+3/-1)
debian.trunk/control (+1/-0)
doc/configdrive/README (+118/-0)
doc/examples/cloud-config-chef-oneiric.txt (+90/-0)
doc/examples/cloud-config-chef.txt (+48/-6)
doc/examples/cloud-config-datasources.txt (+18/-0)
doc/examples/cloud-config.txt (+11/-0)
doc/kernel-cmdline.txt (+48/-0)
doc/nocloud/README (+55/-0)
setup.py (+1/-0)
tests/unittests/test__init__.py (+242/-0)
tests/unittests/test_datasource/test_maas.py (+153/-0)
tests/unittests/test_handler/test_handler_ca_certs.py (+4/-0)
tests/unittests/test_userdata.py (+107/-0)
tests/unittests/test_util.py (+18/-2)
tools/Z99-cloud-locale-test.sh (+92/-0)
tools/run-pylint (+3/-1)
lp:ubuntu/raring/cloud-init
Revision history for this message
Joshua Harlow (harlowja) wrote :

This only seems to affect boto >= 2.5.2

Revision history for this message
Jeremy Deininger (jeremydei) wrote :

I've seen a race condition happening when loading ec2-public keys for the authorized-keys file. The result was no ec2 authorized_key was being populated. (Archlinux, systemd init scripts, python2-boto 2.5.2)

I'll try this patch and report back. Thanks!

Revision history for this message
Joshua Harlow (harlowja) wrote :

Odd, boto 2.5.2 i don't think had the lazy loading stuff, at least from looking at there code @ https://github.com/boto/boto/blob/2.5.2/boto/utils.py#L244 it seems to be doing a depth first like action there...

Revision history for this message
Jeremy Deininger (jeremydei) wrote :

Tried the boto-metadata-fixings branch and it didn't have an effect. What I'm seeing is failure to populate the .ssh/authorized_keys file with the EC2 ssh key. There is no error message. I've added debug logging to my branch but for some reason adding this logging was enough to get the key populated. http://bazaar.launchpad.net/~jeremydei/cloud-init/archlinux/revision/701#cloudinit/sources/__init__.py

Revision history for this message
Joshua Harlow (harlowja) wrote :

Thats totally weird. Let me give u a patch to apply that might also help. Will get u it when I get back to work. It will debug log all loads/writes that are happening. Useful to track down this kind of stuff.

Revision history for this message
Joshua Harlow (harlowja) wrote :

Try this, it might help diagnoise what is being written out...

Revision history for this message
Joshua Harlow (harlowja) wrote :

Anddd.

Scott Moser (smoser)
Changed in cloud-init:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Jeremy Deininger (jeremydei) wrote :

Joshua,
I launched an instance with those patches applied. Here is the log output. Still appears as though it's not writing the key. I also didn't really see it say much about writing the metadata (only the userdata). What do you think?

http://pastie.org/private/4nlikzlqoawmfmb6m2iibq

Revision history for this message
Joshua Harlow (harlowja) wrote :

Really odd, it almost seems like no userdata was fetched.

Is it possible to get into the instance and do a wget/curl on the 169 urls?

U should set a big blob that is being written out to a userdata.i file. I didn't see that.

It'd be interesting to get in the instance and then call this code.

https://code.launchpad.net/~harlowja/cloud-init/boto-metadata-fixings/+merge/131301

The get_userdata and metadata function there. See what it gets...

Revision history for this message
Jeremy Deininger (jeremydei) wrote :

Ok, I messed around with running this code quite a bit. As far as I can tell the metadata just doesn't get populated when using boto 2.5.2. Even after launching and re-running cloud-init, or re-running cloud-init after wipeing the instance data did not ever seem to store the metadata.

Then I upgraded to boto 2.6.0. When using that I had this traceback:
http://pastie.org/5344298

This is while running the branch for boto-metadata-fixings + the extra debug patches.

When I was using 2.5.2 it seemed like whenever i print the value of one of these variables that makes it populate the data. Perhaps there is something with casting to string or etc that causes the population. Otherwise it seems to be blank.

-J

Revision history for this message
Jeremy Deininger (jeremydei) wrote :

Further inspection while using boto 2.6.0 it looks like the _unlazy_dict method is returning 'None'. Here's a paste showing them side by side. The Boto one appears to be returning a fully populated dict ..

http://pastie.org/5347372

Revision history for this message
Joshua Harlow (harlowja) wrote :

Well thats odd. Let me check into why thats happening. Seems like it shouldn't.

Revision history for this message
Joshua Harlow (harlowja) wrote :

Can u try @ https://code.launchpad.net/~harlowja/cloud-init/boto-metadata-fixings

I think I fixed the none case there, bad return, oops.

Revision history for this message
Jeremy Deininger (jeremydei) wrote :

Sweet that works! It populates the authorized_key now. https://code.launchpad.net/~harlowja/cloud-init/boto-metadata-fixings r700 + the debug patches. I am doing more testing here soon without all the debug stuff..

Revision history for this message
Scott Moser (smoser) wrote :

fix-committed in revno 715.

Changed in cloud-init:
status: Triaged → Fix Committed
Revision history for this message
Tomas Karasek (tomas-karasek) wrote :

hi, I am testing 0.7.0 on Scientific Linux CERN 6 (clone of RHEL6), with python-boto-2.5.2-1.el6.noarch. I build the rpm for cloud-init from trunk, revno 725. My isntances live in Openstack Essex. Python is 2.6.6.

My .ssh/authorized_keys are still not populated.

I debugged through sources/__init__.py, function normalize_pubkey_data(pubkey_data). So following text regards the function.
When I print the type of pubkey_data, it's <class 'boto.utils.LazyLoadMetadata'>. When I print the pubkey_data itself, the object gets evaluated and .ssh/authorized_keys are populated properly. In that case the pubkey_data are a dict:

{'tkarasek_keypair': ['ssh-rsa AAAAB3NzaD1yc2EAAAABIwAAAIEAutjuZ/3b+eY3VvKWJPsgkfJqcjKaobgjJRbr/jCwskGUNc8nGdB3dhOJTFTUi/gycrT8CeebB/wwwM4oz3ip/GJjCPaHS8X4vkqWsqgBQZ36cNzfRnqwKWRTMzAI19ekTIsOdjENyeQ+QJoyOFIU9RJAOoepEp/uMD3PH39FLfk= sth@lhost, '']}

Even if I don't print the pubkey_data, it takes the last branch ("if isinstance(pubkey_data, (dict)):"). However in the loop over iteritems of the dict, I get ('tkarasek_keypair', None), i.e. the value of pubkey_data['tkarasek_keypair'] is reported as None. Because of that, empty list is returned from normalize_pubkey_data.

When I change the loop to
for (_keyname, klist) in pubkey_data.items(): # isntead of for (_keyname, klist) in pubkey_data.iteritems()

I get the proper list in klist and my authorized keys are populated. It seems the iterator doesnt work well with the boto lazy metadata.

Would you change the loop from iteritems to items?

Revision history for this message
Joshua Harlow (harlowja) wrote :

Hi tomas, can u post the contents of the boto/utils.py file that u have which should be in site-packages. I was checking https://github.com/boto/boto/blob/2.5.2/boto/utils.py and it doesn't seem they have the lazy load dictionary there so I am wondering if they put it in then removed it and thats why u are seeing it.

I think scott and I have some ideas on how to fix this (ie traverse LazyLoadMetadata and force it to be a python dictionary so that none of its weirdness happens). But more info would be great. Thx!

Revision history for this message
Joshua Harlow (harlowja) wrote :

On my rhel6 box:

[harlowja@followsave ~]$ sudo yum list | grep boto
Password:
python-boto.noarch 2.5.2-1.el6 @epel

[harlowja@followsave ~]$ python
Python 2.6.6 (r266:84292, Aug 28 2012, 10:55:56)
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto.utils as boto_utils
>>> dir(boto_utils)
['AuthSMTPHandler', 'Encoders', 'ISO8601', 'ISO8601_MS', 'LRUCache', 'MIMEBase', 'MIMEMultipart', 'MIMEText', 'Password', 'ShellCommand', 'StringIO', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '_get_instance_metadata', '_hashfn', 'boto', 'canonical_string', 'datetime', 'fetch_file', 'find_class', 'formatdate', 'get_aws_metadata', 'get_instance_metadata', 'get_instance_userdata', 'get_ts', 'get_utf8_value', 'guess_mime_type', 'gzip', 'hashlib', 'imp', 'logging', 'merge_meta', 'mklist', 'notify', 'parse_ts', 'pythonize_name', 'qsa_of_interest', 'retry_url', 'smtplib', 'subprocess', 'tempfile', 'time', 'update_dme', 'urllib', 'urllib2', 'write_mime_multipart']

I'm not seeing the lazy load stuff there, can u do the same, thx.

Revision history for this message
Tomas Karasek (tomas-karasek) wrote :

Hi Joshua,

it seems there's something wrong with the boto rpm in epel. It says it's 2.5.2 but boto/utils.py contains the LazyLoadMetadata which were introduced later. You can see the code in the tarball in the source rpm:
http://dl.fedoraproject.org/pub/epel/6/SRPMS/repoview/python-boto.html
(to unpack the rpm you can "rpm2cpio python-boto-2.5.2-1.el6.src.rpm | cpio -idmv" if you are interested)

So there's some mess in the boto rpm but all that does not really matter from cloud-init point of view right? Can't we just eventually process the LazyLoadMetadata and transform it do dict? And then work with the dict from then on?

As requested, I attach utils.py from my test instance and the boto routine is:

[root@tomk-cava ~]# yum list | grep boto
python-boto.noarch 2.5.2-1.el6 @epel
[root@tomk-cava ~]# python
Python 2.6.6 (r266:84292, Sep 12 2012, 09:26:39)
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto.utils as boto_utils
>>> dir(boto_utils)
['AuthSMTPHandler', 'Encoders', 'ISO8601', 'ISO8601_MS', 'LRUCache', 'LazyLoadMetadata', 'MIMEBase', 'MIMEMultipart', 'MIMEText', 'Password', 'ShellCommand', 'StringIO', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '_end_cap_regex', '_first_cap_regex', '_get_instance_metadata', '_hashfn', '_number_cap_regex', 'base64', 'boto', 'canonical_string', 'compute_md5', 'datetime', 'fetch_file', 'find_class', 'formatdate', 'get_aws_metadata', 'get_instance_metadata', 'get_instance_userdata', 'get_ts', 'get_utf8_value', 'guess_mime_type', 'gzip', 'hashlib', 'imp', 'json', 'logging', 'md5', 'merge_meta', 'mklist', 'notify', 'parse_ts', 'pythonize_name', 'qsa_of_interest', 're', 'retry_url', 'smtplib', 'socket', 'subprocess', 'tempfile', 'time', 'unquote_v', 'update_dme', 'urllib', 'urllib2', 'write_mime_multipart']
>>>

Revision history for this message
Scott Moser (smoser) wrote :

Tomas,
  To avoid complexity in determining when to lazy load, we're just always traversing the dict now.
  http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/729

So, current trunk (which I hope to release later today as 0.7.2) should be good for you.

Scott Moser (smoser)
Changed in cloud-init:
status: Fix Committed → Fix Released
Revision history for this message
Tomas Karasek (tomas-karasek) wrote :

Tested 0.7.1 and it seems to work on my configuration (as listed above).
Thanks a lot Jeremy, Joshua and Scott.

Revision history for this message
James Falcon (falcojr) wrote :

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2310

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.