Ubuntu
juju package

[MIR] juju, txaws, txzookeeper

Bug #912861 reported by Clint Byrum on 2012-01-06

This bug affects 1 person

	Status	Importance	Assigned to
juju (Ubuntu)	Invalid	High	Unassigned
txaws (Ubuntu)	Invalid	High	Unassigned
txzookeeper (Ubuntu)	Invalid	High	Unassigned

Bug Description

This is a MIR for juju, and two of its chief dependencies, txaws and txzookeeper. There will be a separate MIR for zookeeper since it is a substantial project and stands alone by itself.

== Availability ==

juju was called ensemble before, and entered universe during the oneiric dev cycle. txzookeeper is a twisted zookeeper plugin that was created specifically for juju, and also entered universe during the oneiric cycle. txaws has been available since lucid.

== Rationale ==

juju addresses a need not addressed by plain configuration management or "Platform as a service" offerings. Juju allows one to create and share sets of automated tools for managing services called charms. It has been identified as strategically important to help Ubuntu users take advantage of the cloud and also to create their own scale-out clouds.

== Security ==

juju has not had any "CVE"'s reported against it, and has not been recommended for production usage just yet. It does have a well defined security model that relies mostly on SSH and firewalls for transport layer security. The security model has been examined for weaknesses and bugs are open and tracked upstream to address those weaknesses.

https://bugs.launchpad.net/juju/+bugs?field.tag=security

== Quality assurance ==

Juju is a CLI tool and so does require a bit of documentation reading to understand it.

There are no major bugs open in Debian or Ubuntu for any of these packages.

Upstream uses a Test Driven Dev model, and so there is a rich and rigorous test suite which is run on package build and will fail the build if the test suite fails. Packages are built daily using launchpad daily build PPA's and most developers and interested users test these packages so quality remains high. There are also functional tests that are run on each commit upstream here:

http://wtf.labix.org/

txaws has a rich test suite, which is run and failed on during build.

txzookeeper also has a rich test suite, however, it is somewhat dangerous to run as it will completely delete a local zookeeper's contents, so until this bug is fixed, we can't run it on package build:

https://bugs.launchpad.net/ubuntu/+source/txzookeeper/+bug/912508

UI standards:

N/A

== Dependencies ==

These are the non-main deps/build-deps for the 3 libraries mentioned here:

Deps/Build_Deps:
zookeeper - being addressed in a seperate MIR

Recommends:
pydot - Used to produce "dot" output. Can be dropped to Suggests or added to this MIR. It is a very small python library. I think its worthwhile to add it, but would not want to see this MIR blocked because of it, as this is a very small bit of functionality. pydot also depends on pyparsing, which is in universe. pydot does not seem well maintained, having missed *many* upstream releases.

== Standards compliance ==

All 3 packages are very straightforward and use minimal dh7 rules. They have a few lintian warnings, but mostly due to out of date standards or minor formatting issues.

== Maintenance ==

The package is well maintained in Ubuntu directly. The server team is committed to making juju a big part of Ubuntu's cloud strategy. The package is not in Debian just yet, as its ties to Ubuntu are very strong, though at some point we'd like to make it available in Debian once things stop changing so rapidly.

== Background information ==

This MIR is a part of the following blueprint:

https://blueprints.launchpad.net/ubuntu/+spec/servercloud-p-juju-mir

Michael Terry (mterry) on 2012-01-13

Changed in juju (Ubuntu):
assignee:	nobody → Jamie Strandboge (jdstrand)
Changed in txaws (Ubuntu):
assignee:	nobody → Jamie Strandboge (jdstrand)
Changed in txzookeeper (Ubuntu):
assignee:	nobody → Jamie Strandboge (jdstrand)

Dave Walker (davewalker) on 2012-01-31

Changed in juju (Ubuntu):
importance:	Undecided → High
Changed in txaws (Ubuntu):
importance:	Undecided → High
Changed in txzookeeper (Ubuntu):
importance:	Undecided → High

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2012-03-13:

txzookeeper review:
* no CVE history
* no sudo fragments, dbus services, setuid binaries, initscripts or daemons
* lintian clean
* has test suite, but not run (LP: #912508). Wrote QRT script for this for now
* code is very clean with more that 3x more test code than library code. I like that. :)
* build depends are all in main
* while the package is only in Ubuntu, it is supported by the server team and with upstream employed by Canonical

With the above review, txzookeeper looks good for main.

Changed in txzookeeper (Ubuntu):
assignee:	Jamie Strandboge (jdstrand) → nobody
status:	New → Fix Committed

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2012-03-13:

txaws review:
* no CVE history
* no sudo fragments, privileged operations, dbus services, setuid binaries, initscripts or daemons
* has a test suite and it is run in the build and will fail the build on error
* build logs are clean
* lots of binaries without man pages
* build depends are all in main
* while the package is only in Ubuntu, it is supported by the server team and with upstream employed by Canonical
* aws-status: uses gnome-keyring, which is fine, but this doesn't work with Unity (gtk.StatusIcon needs to move to app indicators). There is no documentation. Should be fixed and documented or dropped. Could alternatively leave binary in universe, but it is unusable atm so I'm not sure of the benefit there.

Conditional ACK provided the following is fixed:
* binaries have worthwhile man pages (--help is useful, but aws-status doesn't
have it and at least txaws-discover is out of date)
* aws-status be fixed and documented or dropped. Optionally put this binary in universe.

Changed in txaws (Ubuntu):
assignee:	Jamie Strandboge (jdstrand) → nobody
status:	New → Fix Committed
status:	Fix Committed → In Progress
assignee:	nobody → Clint Byrum (clint-fewbar)

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2012-03-14:

I just noticed that 'juju bootstrap' dies with:
$ juju bootstrap
2012-03-14 08:54:00,190 INFO Bootstrapping environment 'local' (type: local)...
2012-03-14 08:54:00,191 INFO Checking for required packages...
Missing packages apt-cacher-ng
2012-03-14 08:54:01,387 ERROR Missing packages apt-cacher-ng

This dependency is not declared in the packaging though. Also, it seems odd that this is required. Perhaps making it a Suggests and then having juju have a configurable mirror would be good (I know personally I would rather point juju at a mirror than magic with apt-cacher-ng). Some environments may have squid installed also.

Revision history for this message

Clint Byrum (clint-fewbar) wrote on 2012-03-14:

apt-cacher-ng is already a Suggests of juju. Its only necessary for use in the local provider, and the error message is graceful enough that I'm comfortable with it as-is (notice that the other local-only requirements are recommends.. I dropped apt-cacher-ng to Suggests because it is not in main). Agreed that being strict about only using apt-cacher-ng is not the best plan. bug #897645 is open upstream for a more flexible way to specify the proxy to use.

Revision history for this message

Clint Byrum (clint-fewbar) wrote on 2012-03-24:

txaws man pages are generated from help2man now, so its at least more discoverable. I also added indicator support to aws-status, a basic manpage, and a desktop file.

Revision history for this message

Clint Byrum (clint-fewbar) wrote on 2012-03-24:

These fixes, btw, are awaiting release team approval given beta freeze

Changed in txaws (Ubuntu):
assignee:	Clint Byrum (clint-fewbar) → Jamie Strandboge (jdstrand)

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2012-03-26:

FYI, review is mostly complete. Discussing some things with the server team before posting here.

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2012-03-26:

txaws looks good. Thanks! Please feel free to seed.

Changed in txaws (Ubuntu):
assignee:	Jamie Strandboge (jdstrand) → nobody
status:	In Progress → Fix Committed

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2012-03-27:

Download full text (10.9 KiB)

= Review =
Juju is a very flexible system for deploying services based on industry best practices and expertise. It is very capable and can deploy services to multiple providers. As such, though my review took quite a bit of time, it should still be considered a shallow audit. Understanding this, here is my security review:

juju has support for different providers which are simply the types of cloud frameworks it can use. For example, there is an EC2 provider (which also works with OpenStack) and a Local (LXC) provider. More providers are expected. The providers are configured via ~/.juju/environments.yaml on the admin system. juju abstracts out the specifics of working with a provider one environments.yaml is correctly configured. juju admin host stores sensitive information in ~/.juju/environments.yaml. It does not enforce safe permissions currently (LP: #956009).

juju's architecture is such that an admin runs juju commands on her system and they are delivered to a bootstrapping node. The bootstrapping node runs a zookeeper database and has the ability to start and stop units (nodes) and deliver setup code (charms) to the nodes. The nodes execute the charms code as root. In addition to setup code, charms provide other hooks like 'start' and 'stop' which are executed when the service unit is stopped or started. All the hooks run with root permissions. All the nodes share the same database, but there is only one zookeeper leader so nodes should not be able to be elected as a zookeeper leader (see server.* in /etc/zookeeper/conf/zoo.cfg). All nodes currently are able to read and write to the zookeeper database. With the Local provider, zookeeper is started as the user invoking juju (uses a high non-default port), not in a separate bootstrapping node. In all ways I could see, the admin's system is effectively the bootstrapping node with the Local provider.

In terms of network connectivity, juju allows ssh access to all nodes. When the admin deploys a node via a charm, the node's new service is still not available over the network (but is to other nodes in the environment). Only when the service is 'exposed' does the application become available over the network. For example, if an admin deploys mysql and wordpress services, wordpress is only available to the world after the admin uses 'juju expose wordpress'. This is a good design as it allows the admin to verify the configuration, perform updates, etc before it is exposed to the world. Also, in this example, mysql is correctly not exposed to the world. This is all accomplished via security groups in EC2/OpenStack. In the current version of juju, network access is not a problem with the Local provider because zookeeper and the services are all on the libvirt NAT network and not exposed to the world directly. Expose/unexpose doesn't seem to have any meaning with the Local provider as no firewall rules are added via iptables and the service is not accessible from other hosts (besides the admin machine).

There are many problems surrounding zookeeper access. Anyone who can connect zookeeper (ie, all nodes) can see and modify anything in the database. Note that this does not require subvertin...

juju has support for different providers which are simply the types of cloud frameworks it can use. For example, there is an EC2 provider (which also works with OpenStack) and a Local (LXC) provider. More providers are expected. The providers are configured via ~/.juju/environments.yaml on the admin system.  juju abstracts out the specifics of working with a provider one environments.yaml is correctly configured. juju admin host stores sensitive information in ~/.juju/environments.yaml. It does not enforce safe permissions currently (LP: #956009).

juju's architecture is such that an admin runs juju commands on her system and they are delivered to a bootstrapping node. The bootstrapping node runs a zookeeper database and has the ability to start and stop units (nodes) and deliver setup code (charms) to the nodes. The nodes execute the charms code as root. In addition to setup code, charms provide other hooks like 'start' and 'stop' which are executed when the service unit is stopped or started. All the hooks run with root permissions. All the nodes share the same database, but there is only one zookeeper leader so nodes should not be able to be elected as a zookeeper leader (see server.* in /etc/zookeeper/conf/zoo.cfg).  All nodes currently are able to read and write to the zookeeper database. With the Local provider, zookeeper is started as the user invoking juju (uses a high non-default port), not in a separate bootstrapping node. In all ways I could see, the admin's system is effectively the bootstrapping node with the Local provider.

In terms of network connectivity, juju allows ssh access to all nodes. When the admin deploys a node via a charm, the node's new service is still not available over the network (but is to other nodes in the environment). Only when the service is 'exposed' does the application become available over the network.  For example, if an admin deploys mysql and wordpress services, wordpress is only available to the world after the admin uses 'juju expose wordpress'. This is a good design as it allows the admin to verify the configuration, perform updates, etc before it is exposed to the world. Also, in this example, mysql is correctly not exposed to the world. This is all accomplished via security groups in EC2/OpenStack. In the current version of juju, network access is not a problem with the Local provider because zookeeper and the services are all on the libvirt NAT network and not exposed to the world directly. Expose/unexpose doesn't seem to have any meaning with the Local provider as no firewall rules are added via iptables and the service is not accessible from other hosts (besides the admin machine).

There are many problems surrounding zookeeper access. Anyone who can connect zookeeper (ie, all nodes) can see and modify anything in the database. Note that this does not require subverting the juju agent-- all that is required is a network connection to the zookeeper server and standard tools. Some information appears to be rewritten each time (eg, /environments).

While juju uses security groups for network access (thus limiting who can connect to it) for EC2/OpenStack, it would be best if this was explicit in the nodes firewall configuration (which is a requirement for Maas anyway). For example, these ports on the bootstrapping node are visible to other nodes in the environment:
2181/tcp  open  unknown
38830/tcp open  unknown

2181 is for followers to connect to the leader and 38830 is presumably for leader election.

juju uses ssh for communications with the nodes. The specified ssh key on the admin machine is copied to authorized_keys in the 'ubuntu' account on all nodes. The 'ubuntu' account has an entry in /etc/sudoers.d/90-cloudimg-ubuntu which allows full root access without a password. This mirrors Ubuntu's EC2 implementation and is acceptable.

As mentioned, charm code is executed as root. Security-conscious users will need to verify all charms before deployment. Deploying charms from unknown sources is the equivalent of running executables or installing packages from unknown sources and should be avoided. The juju design of deploying hooks from the admin's machine (as opposed to pulling charms onto the bootstrap node) is good because it allows the admin to verify all charm code and track changes locally. That said, charms are cached onto the bootstrapping node when a charm is deployed.

Various upgrade scenarios are documented as not being implemented yet (service upgrades-- https://juju.ubuntu.com/docs/upgrades.html). Charms provide an upgrade hook as well and work is ongoing to improve charm upgrades. While unattended-upgrades is installed, running 'sudo dpkg-reconfigure unattended-upgrades' shows deployed systems are not setup to download and install security updates automatically.

The Local provider is a bit rough around the edges in the version of juju I tested. While it is intended to only be used for local testing, reboots of the admin machine result in zookeeper and nodes not starting (LP: #955576). This is a major usability issue, reflects poorly on juju and could be construed as data loss. Also, since many people are going to want to test juju in virtualized environments, juju needs to not hard code 192.168.122 in its code (LP: #955540). Also, 'juju destroy-environment' currently needs to be run with sudo and the 'data-dir' removed manually. These all combine to show that Local doesn't work as well as the EC2 provider. I had a lot of problems running it in a VM-- it would be good to document how to do this if there are specific steps to perform to make this work right (beyond fixing the above bugs).

There is a lot of various documentation on juju on the internet. The canonical documentation seems to be at https://juju.ubuntu.com/docs/index.html. There are no man pages and finding the correct documentation to use proved difficult.  Because so much has been blogged and written about juju, the conflicting sources can make its initial use difficult. Also, because juju is rapidly changing, much of what is on the internet is out of date. I had several problems using the Local provider in general and configuring environments.yaml appropriately for different providers. I also was not able to easily understand what control-bucket and admin-secret are in the documentation, though I eventually figured it out. It was also difficult to find a high level design overview or security design document.

In terms of packaging, juju looks fine with no surprises (excepting the lack of man pages).

In terms of coding:
 * juju/control/debug_hooks.py makes use of os.system and should probably be converted to subprocess
 * juju/environment/config.py has a TOCTOU race for creating environments.yaml.  This is not a security concern since it is in the user's directory.
 * http:// calls that can be subject to MITM attacks in juju/providers/ec2/utils.py (LP: #965507) and juju/providers/orchestra/cobbler.py and juju/providers/orchestra/files.py I'm assuming the new Maas provider will also be affected as it is using cobbler.
 * txaws does not verify certificates: LP: #781949
 * juju/providers/common/cloudinit.py does not do any input shell metacharacter injection sanitization for:
   - 'branch' in _branch_install_scripts()
   - 'secret', 'provider_type' in _zookeeper_scripts()
   This aren't necessarily a security vulnerability because the admin controls the machines anyway, but it is good form to perform some sort of input sanitizing when writing out scripts in this manner (since they aren't going through something like subprocess.Popen)

= Conclusion =
Unfortunately, there are a number of problems that prevent me from endorsing juju's promotion to main in its current state. However, juju is an important project and I understand there is a lot of working on improving it for 12.04 and that work will continue on it for 12.04.1. Based on conversations with Robbie Williamson, the server team is committing to a two-stage rollout of fixes to be completed by 12.04.1. I have taken the liberty of filing tracking bugs for these issues (see below). As such, I am ok for promotion to main at this time.

== Requirements (12.04) ==
- address coding concerns, above
- make sure in official blog posts, etc, the official documentation is discoverable as much as possible
- explicit ingress rule on non-Local provider bootstrapping node for zookeeper (LP: #966558)
- juju needs to enforce secure permissions on environments.yaml (LP: #956009)
- document best practices for keeping systems up to date (LP: #966563)
- (create/)document charm store review process. This process should include promoting the deploying AppArmor policy in charms (LP: #966566)
- document in release notes the lack of zookeeper ACLs, with an appropriate warning. This should also be mentioned in juju documentation as something being worked on (LP: #966573)
- explicit egress 'owner' rule on non-bootstrapping nodes to require root access to zookeeper (LP: #966577)
- document lack of encryption in the juju environment in documentation and release notes (LP: #966583)
- ensure Local provider properly exposes and unexposes services if this is part of the final release (ie, something akin to security groups for the Local provider such that you must expose services before they are available to the network)
- explicit ingress filtering for Maas nodes which should mimic the EC2/OpenStack security groups such that expose and unexpose work as expected LP: #966584

== Requirements (12.04.1) ==
- fix Local provider bug LP: #955576 (and ideally LP: #955540)
- explicit ingress filtering on non-Local provider bootstrapping node (eg allow ping and 22/tcp from anywhere, but only ping, 22/tcp, and 2181 only from nodes in this environment (LP: #966590) 
- implement zookeeper ACLs (eg admin for writes and something else for juju agent reads. world would be closed off entirely) (LP: #813773)
- if possible, encipher or remove sensitive credentials from zookeeper (LP: #966601)
- document best practices for securing communication between juju nodes and in the case of Maas, also between juju and cobbler. Remove an hard-coded assumptions from juju (eg, http vs https) and ideally add some functionality to make this easier to deploy. Suggestions are stunnel4 or openvpn (LP: #966605)
- man pages for juju commands. Utilize 'make man' in the build to generate them automatically. Also have an overview man page to point to the official documentation and have a brief tutorial for using each supported provider, overviews of how charms work, to trust charms carefully, how to do maintenance, etc. This overview can include an best practices for using the various providers (LP: #966611)
- supply high level design and security document (LP: #966617)

Changed in juju (Ubuntu):
assignee:	Jamie Strandboge (jdstrand) → nobody
status:	New → Fix Committed

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2012-04-27:

#10

I am going to mark this back to 'In Progress'. The server team decided not to pursue juju for main inclusion in 12.04 so I am removing the conditional ACK until the bugs I outlined are fixed.

Changed in juju (Ubuntu):
status:	Fix Committed → In Progress

Revision history for this message

James Page (james-page) wrote on 2014-03-14:

#11

Marking bugs as invalid as this codebase is no longer under MIR

Changed in juju (Ubuntu):
status:	In Progress → Invalid
Changed in txaws (Ubuntu):
status:	Fix Committed → Invalid
Changed in txzookeeper (Ubuntu):
status:	Fix Committed → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntujuju package

[MIR] juju, txaws, txzookeeper

Bug Description

Other bug subscribers

Remote bug watches

Ubuntu
juju package