Failed Deploy 'mongodb' service, failed: not running with --replSet, hook "replica-set-relation-changed" failed

Bug #1513094 reported by mahmoh
40
This bug affects 6 people
Affects Status Importance Assigned to Milestone
mongodb (Juju Charms Collection)
Fix Released
High
Mario Splivalo

Bug Description

Failed Deploy 'mongodb' service

unit-mongodb-0[1996]: 2015-11-04 04:02:33 INFO unit.mongodb/0.juju-log server.go:254 replica-set:6: init_replset: OperationFailure: command SON([('replSetGetStatus', 1)]) failed: not running with --replSet
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 Traceback (most recent call last):
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-mongodb-0/charm/hooks/replica-set-relation-changed", line 1662, in <module>
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 hooks.execute(sys.argv)
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-mongodb-0/charm/hooks/charmhelpers/core/hookenv.py", line 672, in execute
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 self._hooks[hook_name]()
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-mongodb-0/charm/hooks/replica-set-relation-changed", line 1196, in replica_set_relation_changed
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 init_replset()
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-mongodb-0/charm/hooks/replica-set-relation-changed", line 467, in init_replset
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 r = run_admin_command(c, 'replSetGetStatus')
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-mongodb-0/charm/hooks/replica-set-relation-changed", line 499, in run_admin_command
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 return client.admin.command(cmdstr)
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 File "/usr/lib/python2.7/dist-packages/pymongo/database.py", line 396, in command
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 msg, allowable_errors)
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 File "/usr/lib/python2.7/dist-packages/pymongo/helpers.py", line 147, in _check_command_response
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 raise OperationFailure(msg % errmsg, code)
unit-mongodb-0[1996]: 2015-11-04 04:02:35 INFO unit.mongodb/0.replica-set-relation-changed logger.go:40 pymongo.errors.OperationFailure: command SON([('replSetGetStatus', 1)]) failed: not running with --replSet
unit-mongodb-0[1996]: 2015-11-04 04:02:35 ERROR juju.worker.uniter.operation runhook.go:103 hook "replica-set-relation-changed" failed: exit status 1

Easier to read paste: http://paste.ubuntu.com/13102315/

Related branches

Revision history for this message
mahmoh (mahmoh) wrote :
David Britton (dpb)
description: updated
description: updated
information type: Proprietary → Public
no longer affects: landscape
mahmoh (mahmoh)
summary: - Failed Deploy 'mongodb' service
+ Failed Deploy 'mongodb' service, failed: not running with --replSet,
+ hook "replica-set-relation-changed" failed
Revision history for this message
Brad Marshall (brad-marshall) wrote :

This occurs on Xenial (and perhaps earlier) because the charm takes the simple approach of replacing the string "--" with "-- --replSet $set" to add the replSet option in. Unfortunately in Xenial, this is no longer sufficient - it changes the options to numactl, which doesn't know anything about this option, and doesn't pass it down to mongodb.

tags: added: landscape
tags: added: kanban-cross-team
Revision history for this message
Billy Olsen (billy-olsen) wrote :

So it appears that the mongodb charm will either set the master = True or slave = True in /etc/mongodb.conf file. Starting with mongodb 2.6, this commit (https://github.com/mongodb/mongo/commit/f9841e0136d9f7a667662581cda9ce9ddfbf20d8) makes the master or slave option incompatible with the replication.replSet option.

Changed in mongodb (Juju Charms Collection):
status: New → Triaged
importance: Undecided → High
assignee: nobody → Billy Olsen (billy-olsen)
Changed in mongodb (Juju Charms Collection):
status: Triaged → In Progress
Changed in mongodb (Juju Charms Collection):
assignee: Billy Olsen (billy-olsen) → Mario Splivalo (mariosplivalo)
tags: added: canonical-bootstack
Revision history for this message
Benjamin Kaehne (ben-kaehne) wrote :

This bug is also affecting me. (xenial, systemd)

I notice the sytemd unit file is not populated with any data/config pertaining to replicasets etc.

tags: removed: kanban-cross-team
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

This is being fixed in trusty, and will work in xenial too. The --replSet is removed from the upstart script and put into the mongod.conf (where it belongs).

Revision history for this message
Jay R. Wren (evarlast) wrote :

This appears to also error on replica-set-relation-broken. For some reason this hook is run even if the charm was never part of a replica-set, when removing the charm.

Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

The patched charm can be found in charmstore, in my personal namespace:

https://jujucharms.com/u/mariosplivalo/mongodb

Use this with word of caution as sharding support is non functional currently.

The 'proper' xenial charm is being developend, and the progress can be tracked here:

https://github.com/marcoceppi/layer-mongodb/

Revision history for this message
Jay R. Wren (evarlast) wrote :

2016-11-17 18:36:46 INFO juju-log replica-set:5: replica_set_relation_broken-start
2016-11-17 18:36:46 DEBUG juju-log replica-set:5: am_i_primary: OperationError: command SON([('replSetGetStatus', 1)]) failed: not running with --replSet
2016-11-17 18:36:48 INFO replica-set-relation-broken Traceback (most recent call last):
2016-11-17 18:36:48 INFO replica-set-relation-broken File "/var/lib/juju/agents/unit-mongodb-1/charm/hooks/replica-set-relation-broken", line 1684, in <module>
2016-11-17 18:36:48 INFO replica-set-relation-broken hooks.execute(sys.argv)
2016-11-17 18:36:48 INFO replica-set-relation-broken File "/var/lib/juju/agents/unit-mongodb-1/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-11-17 18:36:48 INFO replica-set-relation-broken self._hooks[hook_name]()
2016-11-17 18:36:48 INFO replica-set-relation-broken File "/var/lib/juju/agents/unit-mongodb-1/charm/hooks/replica-set-relation-broken", line 1266, in replica_set_relation_broken
2016-11-17 18:36:48 INFO replica-set-relation-broken if am_i_primary():
2016-11-17 18:36:48 INFO replica-set-relation-broken File "/var/lib/juju/agents/unit-mongodb-1/charm/hooks/replica-set-relation-broken", line 1176, in am_i_primary
2016-11-17 18:36:48 INFO replica-set-relation-broken r = run_admin_command(c, 'replSetGetStatus')
2016-11-17 18:36:48 INFO replica-set-relation-broken File "/var/lib/juju/agents/unit-mongodb-1/charm/hooks/replica-set-relation-broken", line 506, in run_admin_command
2016-11-17 18:36:48 INFO replica-set-relation-broken return client.admin.command(cmdstr)
2016-11-17 18:36:48 INFO replica-set-relation-broken File "/usr/lib/python2.7/dist-packages/pymongo/database.py", line 396, in command
2016-11-17 18:36:48 INFO replica-set-relation-broken msg, allowable_errors)
2016-11-17 18:36:48 INFO replica-set-relation-broken File "/usr/lib/python2.7/dist-packages/pymongo/helpers.py", line 147, in _check_command_response
2016-11-17 18:36:48 INFO replica-set-relation-broken raise OperationFailure(msg % errmsg, code)
2016-11-17 18:36:48 INFO replica-set-relation-broken pymongo.errors.OperationFailure: command SON([('replSetGetStatus', 1)]) failed: not running with --replSet
2016-11-17 18:36:48 ERROR juju.worker.uniter.operation runhook.go:107 hook "replica-set-relation-broken" failed: exit status 1

Changed in mongodb (Juju Charms Collection):
status: In Progress → Fix Committed
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

The xenial charm has been released into the charmstore.

Please let me know if you still experience this issue.

Changed in mongodb (Juju Charms Collection):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.