Rabbit reports "file descriptor limit alarm set", does not accept connections
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
rabbitmq-server (Ubuntu) |
Expired
|
Undecided
|
Unassigned |
Bug Description
We had an incident where rabbitmq-server would stop accepting connections, with the following message in the log:
=WARNING REPORT==== 24-Nov-
file descriptor limit alarm set.
*******
*** New connections will not be accepted until this alarm clears ***
*******
However, when checking the fd limits these are set quite high:
root@juju-
# Generated by juju
# bump ulimit so rabbit can support lots of connections
ulimit -n 65536
root@juju-
LimitNOFILE=65536
On the other hand, checking fd usage via lsof, `lsof -u rabbitmq` would only report abt. 900 open fds
After bouncing rabbitmq it resumed accepting connects
Versions:
rabbitmq-server 3.5.7-1ubuntu0.
Ubuntu 16.04.5 LTS
Changed in rabbitmq-server (Ubuntu): | |
status: | Expired → New |
Thank you for taking the time to file a bug report.
Would you be able to come up with a way for us to reproduce this bug? I understand that it must be happening in a production environment where thousands of connections are being made, but maybe we can come up with a way to reproduce it locally as well.
Does this bug happen often, or is it the first time it happened?
I tried finding some similar cases on the internet, but what I found were reports of real scenarios where increasing the fd limit solved the problem. Curiously, one of the solutions was to add the LimitNOFILE directive to the systemd unit file, which is already present in your case.
I'm subscribing the Ubuntu Server team to this bug so that we can follow it closely.