fix channel_termination_timeout

Bug #1936881 reported by Seyeong Kim
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
rabbitmq-server (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
New
Undecided
Unassigned

Bug Description

[Impact]

Bionic, Openstack

The customer reported below issue

And there are queues which has unconsumed msgs. so wasn't able to create instance.

they was able to fix this by rebooting nova-compute node.

=CRASH REPORT==== 2-Jul-2021::04:30:57 ===
crasher:
initial call: rabbit_reader:init/4
pid: <0.17293.17>
registered_name: []
exception exit: channel_termination_timeout
in function rabbit_reader:wait_for_channel_termination/3 (src/rabbit_reader.erl, line 800)
in call from rabbit_reader:send_error_on_channel0_and_close/4 (src/rabbit_reader.erl, line 1548)
in call from rabbit_reader:terminate/2 (src/rabbit_reader.erl, line 642)
in call from rabbit_reader:handle_other/2 (src/rabbit_reader.erl, line 567)
in call from rabbit_reader:mainloop/4 (src/rabbit_reader.erl, line 529)
in call from rabbit_reader:run/1 (src/rabbit_reader.erl, line 454)
in call from rabbit_reader:start_connection/4 (src/rabbit_reader.erl, line 390)

There is upstream patch and they also doesn't have reliable reproducer.

https://github.com/rabbitmq/rabbitmq-server/pull/1550
https://github.com/rabbitmq/rabbitmq-server/issues/544

[Test Case]
- not able to reproduce this.
- I made test pkg with this patch, and test openstack env worked fine. but need review
- - https://launchpad.net/~seyeongkim/+archive/ubuntu/sf314324/

[Where problems could occur]
As this patch is for rabbitmq-server, rabbitmq-server should be restarted and messaging between components may have problem

[Others]

Tags: sts
Seyeong Kim (seyeongkim)
tags: added: sts
description: updated
Seyeong Kim (seyeongkim)
Changed in rabbitmq-server (Ubuntu):
status: New → Fix Released
Seyeong Kim (seyeongkim)
description: updated
description: updated
Seyeong Kim (seyeongkim)
description: updated
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thanks for the report and the patch, Seyeong.

I understand that these rabbitmq-server bugs are indeed hard to reproduce; this is unfortunately not the first time it happens :-/.

I looked at the debdiff you posted and I have a few comments about it:

- Since the patch comes from upstream (which is great, btw), it is common practice to add DEP-3 headers to it.

- Given that it was not possible to come up with a local reproducer for this issue, do you think the customer would be willing to try a PPA build of rabbitmq-server containing this patch? I think this is the "next best thing" in order to at least verify that this backport works for them.

- As a side note: rabbitmq-server is being sync'ed by git-ubuntu, which means that you can "git ubuntu clone rabbitmq-server", work on the proper branch (ubuntu/bionic-devel, in this case), push it to your LP namespace and create an MP off of it. It's not required, but may be easier to review/discuss. Up to you :-).

Thanks.

Revision history for this message
Seyeong Kim (seyeongkim) wrote :
Revision history for this message
Seyeong Kim (seyeongkim) wrote :

@Sergio

Thanks for your review, I've uploaded debdiff with dep3(sorry I forgot)

Let me ask the customer if it is possible to test this ppa,

Thanks

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

I took a quick look at the debdiff and LGTM (I did not try to build it and run tests), but I'll let Sergio take a look at this and proceed with the sponsoring.

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thanks, Seyeong and Lucas.

Seyeong, no problem at all, and thanks for the quick follow-up! I will wait until you have news from the customer before we proceed with the upload.

FWIW, I can't check the patch right now because the URL is 503'ing here (internal LP problem, most likely). But I remember that the patch was pretty much OK aside from DEP-3 issue, so if that's been fixed then it should be good to go.

Cheers.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.