Upgrade to RabbitMQ 3.6.10 causes beam lockup in clustered deployment

Attaching web.yaml, a heat template known to trigger this event.

Diff between 18.02 and 18.05 rmq charm:

http://paste.ubuntu.com/p/K2RT3QvGkQ/

Logs from /var/log/rabbitmq might be useful here

Status changed to 'Confirmed' because the bug affects multiple users.

Adding that the config "management_plugin=true" causes the crashing to happen with the web.yaml stack, while having it false does not cause the crashing.

Now that is a useful nugget of information.

for people being hit by this bug, it would be helpful if you could generate a core file while rabbit is still stuck, this may give us some insights of where the process is looping.

$ sudo gcore -o lp1783203.core $PID

We are also seeing this issue after upgrading OpenStack from Pike to Queens. It only seems to affect our larger setups, it wasn't seen during testing on our staging setup. The good thing is that I can confirm that disabling the management plugin seems to avoid the issue. We also have a core dump, but since it is from a production environment, I cannot share the content. However, the trace from thread 1 looks like

#0 0x00007fb09cd565d3 in select () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000000000563c00 in erts_sys_main_thread ()
#2 0x0000000000469860 in erl_start ()
#3 0x000000000042f389 in main ()

and the other threads all seem to be at

#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00000000005c15ed in ethr_event_wait ()
#2 0x000000000051e0a5 in ?? ()
#3 0x00000000005c0da5 in ?? ()
#4 0x00007fb09d2326ba in start_thread (arg=0x7fb098ce6700) at pthread_create.c:333
#5 0x00007fb09cd6041d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Binary is /usr/lib/erlang/erts-7.3/bin/beam.smp from erlang-base 1:18.3-dfsg-1ubuntu3.1.

Please let me know if you need further information, you can also reach me as "frickler" in #ubuntu-server.

Forgot to mention: The log in /<email address hidden> just stops when the server hangs, no output until after it is being restarted.

The latest release in the 3.6.x series seems to be 3.6.16. Almost all releases since 3.6.10 list changes to the management plugin, but I didn't spot anything obvious. Here are the release notes for each:

https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_6_16
https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_6_15
https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_6_14
https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_6_13
https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_6_12
https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_6_11

Maybe this in 3.6.12? https://github.com/rabbitmq/rabbitmq-server/issues/1346

Can anybody more familiar with the problem take a quick peek at the release notes to see if something about this bug jumps out?

From the commit message https://github.com/rabbitmq/rabbitmq-server/pull/1431 "Avoid infinite loop when dropping entries in the GM" in 3.6.15 sounds interesting, as we do seem to see some kind of infinite loop here. I don't have an explanation though as to why GM would be interacting with the management plugin in the way we see.

There isn't a whole lot of details about node state here but GM (a multicast module) has nothing to do with the management plugin.

According to the above comments this is on Erlang 18.3 which has known bugs that stop any activity on a node that had accepted any TCP connections (including HTTP requests) [1][2]. They were reported by team RabbitMQ in mid-2017 and fixed shortly after. Erlang 19.3.6.4 is the minimum supported version for RabbitMQ 3.6.16 primarily because of those issues.

Somewhat related: RabbitMQ 3.6.x is out of support [3][4] and since January 2018 [4], Erlang 19.3.6.4 is the minimum supported version even for 3.6.x.

1. https://bugs.erlang.org/browse/ERL-430
2. https://bugs.erlang.org/browse/ERL-448
3. https://www.rabbitmq.com/which-erlang.html#old-timers
4. http://www.rabbitmq.com/changelog.html

I mostly failed to explain the impact of the above Erlang bugs.

Due to ERl-430 and ERL-448 above, nodes that had any TCP connections open (possibly from HTTP requests) could fail to stop (shut down), accept new TCP connections (including from HTTP clients, RabbitMQ CLI tools) and respond on already open TCP connections.

All of those behaviors are particularly problematic during upgrades. In fact, it was discovered as part of an upgrade of the RabbitMQ BOSH release [1].

1. https://github.com/pivotal-cf/cf-rabbitmq-release

raising erlang bug task; I think we need to address the two ERL-* issues in the xenial erlang version as identified in #13.

Triaged the charm task until we can validate the fixes against the erlang/rabbit bits

We should revisit if this is still an issue with the Focal version of RabbitMQ.