Nodes can become unstable under heavy disk IO pressure from an external process

[wangyusheng1985]

Describe the bug

Hi RabbitMQ Team,

We have observed an issue across multiple production environments where the RabbitMQ server process crashes or becomes unstable when the underlying disk experiences extremely high IO pressure.
This situation often occurs during large-scale external database backup operations, which cause heavy disk read/write contention.

Observed behavior:

• RabbitMQ nodes crash or become unresponsive.

• Logs show sudden loss of RA (Raft) processes.

Connections are dropped unexpectedly.

Possible cause:
Based on repeated observations, it appears that severe disk IO latency or temporary disk unavailability can trigger failures in internal components, particularly those related to quorum and persistence mechanisms.
Given RabbitMQ’s reliance on disk for durability and cluster metadata, it seems sensitive to underlying IO performance degradation.

Notes:

• RabbitMQ version: 3.13.6

• IO pressure is external (caused by database backup processes).

• This behavior is not always reliably reproducible, but has been observed multiple times under similar conditions.

Thank you for maintaining RabbitMQ and continuing to make it better!

Best regards

Reproduction steps

...

Expected behavior

rabbitmq.log.zip

Additional context

No response

[michaelklishin]

@wangyusheng1985 RabbitMQ 3.13.x and 4.0.x are out of community support.

I won't go over 176 MiB logs dumped on me without any details but it immediately visible that this node does not even have a chance to boot, multiple CQ and QQ-related components immediately begin running into exceptions, including CQ message store and index recovery.

RabbitMQ nodes should not be co-hosted with other I/O heavy tools. Every Raft-based feature in modern RabbitMQ is I/O heavy (streams in particular), and latency spikes will result in exceptions, leader elections and such.

4.1.0 has somewhat relevant improvements but nothing changes fundamentally around the fact that quorum queues, streams, Khepri, and even classic queues are I/O heavy.