v3.13 seems to crash on my RabbitMQ v3.13

[joesdu]

Describe the bug

2024-03-12 22:47:55.572699+08:00 [info] <0.250.0> Running boot step rabbit_delayed_message defined by app rabbitmq_delayed_message_exchange
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0>
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0> BOOT FAILED
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0> ===========
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0> Exception during startup:
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0>
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0> exit:{aborted,{node_not_running,rabbit@EQ12}}
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0>
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0> mnesia:abort/1, line 360
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0> rabbit_table:wait/3, line 115
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0> rabbit_boot_steps:-run_step/2-lc$^0/1-0-/2, line 51
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0> rabbit_boot_steps:run_step/2, line 58
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0> rabbit_boot_steps:-run_boot_steps/1-lc$^0/1-0-/1, line 22
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0> rabbit_boot_steps:run_boot_steps/1, line 23
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0> rabbit:start/2, line 974
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0> application_master:start_it_old/4, line 293
2024-03-12 22:47:55.581429+08:00 [error] <0.250.0>
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> crasher:
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> initial call: application_master:init/4
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> pid: <0.249.0>
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> registered_name: []
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> exception exit: {{aborted,{node_not_running,rabbit@EQ12}},
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> {rabbit,start,[normal,[]]}}
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> in function application_master:init/4 (application_master.erl, line 142)
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> ancestors: [<0.248.0>]
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> message_queue_len: 1
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> messages: [{'EXIT',<0.250.0>,normal}]
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> links: [<0.248.0>,<0.44.0>]
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> dictionary: []
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> trap_exit: true
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> status: running
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> heap_size: 2586
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> stack_size: 28
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> reductions: 184
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0> neighbours:
2024-03-12 22:47:56.582065+08:00 [error] <0.249.0>

Reproduction steps

1. sudo cp /home/tom/rabbitmq_delayed_message_exchange-3.13.0.ez /usr/lib/rabbitmq/lib/rabbitmq_server-3.13.0/plugins/
2. sudo rabbitmq-plugins enable rabbitmq_delayed_message_exchange

Enabling plugins on node rabbit@EQ12:
rabbitmq_delayed_message_exchange
The following plugins have been configured:
rabbitmq_delayed_message_exchange
rabbitmq_management
rabbitmq_management_agent
rabbitmq_web_dispatch
Applying plugin configuration to rabbit@EQ12...
Stack trace:

** (CaseClauseError) no case clause matching: {:badrpc, {:EXIT, {:aborted, {:node_not_running, :rabbit@EQ12}}}}
(rabbitmqctl 3.13.0-dev) lib/rabbitmq/cli/plugins/plugins_helpers.ex:215: RabbitMQ.CLI.Plugins.Helpers.update_enabled_plugins/2
(rabbitmqctl 3.13.0-dev) lib/rabbitmq/cli/plugins/plugins_helpers.ex:112: RabbitMQ.CLI.Plugins.Helpers.update_enabled_plugins/4
(rabbitmqctl 3.13.0-dev) lib/rabbitmq/cli/plugins/commands/enable_command.ex:127: anonymous fn/6 in RabbitMQ.CLI.Plugins.Commands.EnableCommand.do_run/2
(elixir 1.15.4) lib/stream.ex:1530: anonymous fn/2 in Stream.iterate/2
(elixir 1.15.4) lib/stream.ex:1757: Stream.do_unfold/4
(elixir 1.15.4) lib/stream.ex:1828: Enumerable.Stream.do_each/4
(elixir 1.15.4) lib/stream.ex:1052: Stream.do_transform_inner_enum/7
(elixir 1.15.4) lib/stream.ex:1828: Enumerable.Stream.do_each/4

{:case_clause, {:badrpc, {:EXIT, {:aborted, {:node_not_running, :rabbit@EQ12}}}}}

Expected behavior

When I copy the plugin to the plugin directory and enable the plugin, I get an error that the RabbitMQ service cannot be started successfully, and after closing the plugin, the service will run correctly.

Additional context

No response

[lukebakken]

I can't reproduce what you report. If you can provide a reliable way to reproduce your issue, I will re-open this GitHub issue.

Please ensure you are using Erlang 26.

Please see the attached transcript, where I download RabbitMQ 3.13, this plugin, and everything starts correctly:

rabbitmq-delayed-message-exchange-268.txt

[joesdu]

I just confirmed, it's indeed 26.x I'll test it again tomorrow.

[michaelklishin]

According to the stack trace, the plugin tries to await a table on node rabbit@EQ12 that not is not running. I cannot think of a specific scenario that would lead to this but it must involve an upgrade.

In any case,

sudo cp /home/tom/rabbitmq_delayed_message_exchange-3.13.0.ez /usr/lib/rabbitmq/lib/rabbitmq_server-3.13.0/plugins/

looks completely wrong. As the Plugin troubleshooting guide suggests, plugin file permission is a really common reason for obscure errors that stem from node's inability to load a root-owned file because RabbitMQ nodes usually do not and should not run as root.

@joesdu unless you are running RabbitMQ as root, your installation steps are wrong.

[joesdu]

According to the stack trace, the plugin tries to await a table on node rabbit@EQ12 that not is not running. I cannot think of a specific scenario that would lead to this but it must involve an upgrade.

In any case,

sudo cp /home/tom/rabbitmq_delayed_message_exchange-3.13.0.ez /usr/lib/rabbitmq/lib/rabbitmq_server-3.13.0/plugins/

looks completely wrong. As the Plugin troubleshooting guide suggests, plugin file permission is a really common reason for obscure errors that stem from node's inability to load a root-owned file because RabbitMQ nodes usually do not and should not run as root.

@joesdu unless you are running RabbitMQ as root, your installation steps are wrong.

thx，I'll test it again tomorrow.

[joesdu]

I tried to modify the permissions and other operations, but I can not start normally, but I can load the plugin normally after deploying through docker compose, it may be that there is something wrong with my environment.

[pnchen]

I fixed With this
#271 (comment)

[joesdu]

Thanks!