The re-published message to a delayed excahnge is not routing to queue

[shamim40]

Describe the bug

I'm using RabbitMqserver 3.8.2 and Erlang 22.2.7 under ubuntu 20.

I have an x-delayed-message exchange set to send messages in at most 5 minutes to a queue.
I couldn't find a way to view the messages stored in exchange/Mnesia.

In a high volume production environment, The message published to a delayed exchange is not routing to queue. The header has x-delay time 300 secs. When the queue is empty in a particular time frame, still the messages stuck in delayed queue.

No messages stuck in test environment, it flows through the delayed exchange to queue. Same installation package for both test vm as well as prod vm

Type | x-delayed-message
Features |   arguments:    x-delayed-type:    fanout
               |   durable:    true
               |  messages delayed:   290

Reproduction steps

Using following python program which publishes the messages:

def send(self, properties, message={}, delay_ms=0):
        if delay_ms > 0:
            delay = delay_ms
        else:
            delay = DEFAULT_DELAY
            if 'headers' in str(properties):
                x_delay = properties.headers.get('x-delay')
                if x_delay:
                    delay = int(x_delay)
                    delay = delay * 2 if (delay and delay > 0) else DEFAULT_DELAY

        if delay > MAX_DELAY:
            delay = MAX_DELAY

        try:
            LOGGER.debug("sending message: <{}>...".format(message))
            self.channel.basic_publish(exchange=self.exchange_name, routing_key=self.routing_key, body=message, properties=pika.BasicProperties(delivery_mode=2, headers={'x-delay': str(delay)}))
        except AMQPConnectionError as err:
            LOGGER.error("In send() - Connection was closed: {}, retrying...".format(err))
            self.connect()
            self.channel.basic_publish(exchange=self.exchange_name, routing_key=self.routing_key, body=message, properties=pika.BasicProperties(delivery_mode=2, headers={'x-delay': str(delay)}))

Expected behavior

The messages should flow from delayed exchange to queue.

Additional context

No response

[lukebakken]

Please note that RabbitMQ 3.8.x is completely out of support.

Please attempt to reproduce this issue using a supported version of RabbitMQ. If you can do so, you must make it easy for us to reproduce. In this case, a docker compose project that can be run to demonstrate the issue would be ideal.

Right now your code doesn't even run as-is, and you have given NO indication of how you have configured RabbitMQ and set up your environment. You are asking us to spend time guessing how to reproduce your issue, rather than providing an easy way to run an environment where we can focus on diagnosing and fixing the issue.

[shamim40]

Is there a way to view the messages stored in exchange or Mnesia?

Response to your statements:

"Please attempt to reproduce this issue.....If you can do so, you must make it easy for us to reproduce."
As I said above, the delayed exchange routes messages to the queue with no issue in test environment with 3.8.x or supported version. This is happening only in one of the production (high volume) systems. I am NOT able to reproduce with supported rabbitmq version as well.

"Right now your code doesn't even run as-is,"
My apology, i shouldn't put this in "Reproduction steps" section. This is just the publisher; it's showing how the delayed time is set in header.

"you have given NO indication of how you have configured RabbitMQ "

Not much customization. Just the delayed plugin and mnesia location change as below:

/etc/rabbitmq/rabbitmq-env.conf:
RABBITMQ_MNESIA_BASE=/opt/rabbitmq-mnesia

/etc/rabbitmq/rabbitmq.conf

listeners.tcp.default=5672
heartbeat=900

Plugins
/usr/lib/rabbitmq/lib/rabbitmq_server-/plugins/rabbitmq_delayed_message_exchange-.ez

Following commands were used in an installer script:

rabbitmq-plugins enable rabbitmq_delayed_message_exchange

fanout_exchange/queue names: uca_instance_updated_fx => uca_db_update_q (no binding key)

rabbitmqadmin --user=$user --password=$password declare exchange name=uca_instance_updated_fx durable=true type=x-delayed-message arguments='{"x-delayed-type": "fanout"}'

rabbitmqadmin --user=$user --password=$password declare queue name=uca_db_update_q durable=true

rabbitmqadmin --user=$user --password=$password declare binding source="uca_instance_updated_fx " destination_type="queue" destination="uca_db_update_q "

[michaelklishin]

In more modern versions there is an exchange-level metric that shows the counter of the number of delayed messages.

Let's not forget that "out of support" means "out of support'.

[shamim40]

Yes, i can see the counter at exchange-level as below:

Type | x-delayed-message
Features | arguments: x-delayed-type: fanout
| durable: true
| messages delayed: 290

Is there a way to view the full message (stuck in delayed exchange) to troubleshoot?

[michaelklishin]

Only using ets:tab2list on the table that stores them via rabbitmqctl eval.

I repeat, "out of support" means "out of support".