not okForStorage error when deploying local charm

not okForStorage seems to be MongoDB's way of telling you it doesn't accept the syntax of your document.
The ones that come up while googling are having either "." or "$" in your field names (they are allowed in values).

The problem was down to having bad character in the config.yaml field names. In this case a full stop "."

e.g:

options:
  db.path:
    default: "/var/lib/db"
    type: string
    description: The path where the data files will be kept.

We should try to teach something to warn about this sort of configuration,
since Mongo won't allow keys to have "." or "$" in them.

On Wed, Apr 16, 2014 at 11:36 AM, Matthew Williams <
<email address hidden>> wrote:

> The problem was down to having bad character in the config.yaml field
> names. In this case a full stop "."
>
> e.g:
>
> options:
> db.path:
> default: "/var/lib/db"
> type: string
> description: The path where the data files will be kept.
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> https://bugs.launchpad.net/bugs/1308146
>
> Title:
> not okForStorage error when deploying local charm
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1308146/+subscriptions
>

I just hit what looks like this bug:
ERROR error uploading charm: cannot update uploaded charm in state: not okForStorage

However "." in the charm option names are not a problem in another environment (deployed from the same server). For example I have "url.base" option and it works fine.

The only difference between the two is that one which works was deployed with older juju version, I can't tell which one exactly but likely 1.16.X

This was brought up today by James Westby:

We have an existing environment that we are trying to redeploy to a new
openstack deployment. One of the charms makes extensive use of dots in
the config keys, and is getting a "not okForStorage" error when trying
to deploy.

It sounds very much like
https://bugs.launchpad.net/juju-core/+bug/1308146.

We could rewrite the charm to avoid the dots, however:

  - Doing a smooth transition would be very difficult.
  - The charm makes extensive use of these, in particular the config
    file that they are serialized to uses them as well, so there would
    have to be a lot of code added to transform the keys in the charm.

It's not exactly clear from the bug log when this regression occured,
and the bug suggests it may only affect local charms.

Could someone look at this and tell us:

  - Whether it is considered a regression that will be fixed, or whether
    this is now a restriction on charms?
  - If it is a regression when we can expect a fix, and whether it will
    be backported to 1.18?

Otherwise I will get on with modifying the charm to avoid the dots, with
all the hassles that brings.

The charm that is having the problem is lp:~ubuntuone-pqm-team/canonical-is-charms/ubuntuone-servers

We do use the configuration names as a key in mongo, but we escape any periods or dollar signs before they are saved to the database, specifically to avoid this problem, so it is odd that this is still happening.

So it looks like 1.16 and earlier didn't use the upload HTTP api provided by the state server? That's presumably
why we are seeing this now? And if there are some codepaths that do the escaping it is because the HTTP
API doesn't, and so that's the regression?

The bug is trivial to reproduce, I made what is essentially an empty charm, with a config.yaml that looks like this:

options:
  foo.bar:
    type: int
    default: 3
    description: Number of foo.bars.

And it causes the not okForStorage error.

you can reproduce it by doing juju deploy cs:~natefinch/testcharm

The problem is even though we *do* escape $ and . for service settings stored in the settings collection, we don't do the same for charm config stored in the charms collection (either uploaded locally or from the charm store). I'm afraid this has been an issue ever since I introduced API for uploading charms, so I'm assigning this to myself and will propose a fix which does proper escaping.

Proposed fix for review at http://reviews.vapour.ws/r/55/

Fix landed in trunk in https://github.com/juju/juju/pull/786.
Backported to 1.20, proposed as https://github.com/juju/juju/pull/790, approved and pending to land (bot issues).
Working on a backport for 1.18 as well, as it's the release we have in trusty.

All fixes and backports landed where they belong.

We don't have any facility to test a juju 1.18.5. We can checkout an old version of juju release scripts from when 1.18.4 and attempt to create a release tarball and source package. There is no guarantee this will work because developers can and have moved deps in Lp and github. When CI detects this, we ask the developers to put the deps back. Since the release of 1.18.4, juju moved most of its packages to github.