Background: Recently on-line MQ consumer process ok, but rabbitmq console shows no consumption process, resulting in MQ queue message accumulation, previously directly restart mq, this decision to explore the reasons
Operating time-consuming Daemons
For business reasons, every time 30 W records are imported, 500 records are packed into the mq queue in batches. When consuming, we need to look up tables and interpolation libraries, which takes a long time to process. We use php-amqp libraries. The code is very simple.
$connection = new AMQPStreamConnection(HOST, PORT, USER, PASS, ...);
$channel = $connection->channel();
$channel->queue_declare($queue, false, true, false, false);
$channel->basic_consume($queue, $consumerTag, false, false, false, false, 'consumeLogic');
function consumeLogic(AMQPMessage $message): void
{
// Consumption logic
}
while (count($channel->callbacks)) {
$channel->wait();
}mq heartbeat
1. Rabbit MQ uses heartbeat mechanism to maintain the connection. In normal scenarios, the client expects to inform the server of its own survival by sending heartbeat packets. If the server sends heartbeat twice in a row and the client does not respond, the server will disconnect from the client. The heartbeat interval can be set at each connection.
2. Because php is a synchronous language, it can't send heartbeat packets continuously when running time-consuming tasks in the background. At this point, the server will disconnect, and the client will only find that the queue has been disconnected if it continues to use it.
How rabbitmq handles heartbeat
By reading the source code of the library, we find that it is through the method AbstractIO::check_heartbeat(), which is called every time you use a connection, such as AMQPChannel::basic_consume(),AMQPChannel::basic_consume(),AMQPChannel::basic_consume(), AMQPChannel::basic_consume().
If the heartbeat interval is set, the check_heartbeat() method monitors the time from the last connection. If the client ignores two heartbeats, it will automatically reconnect, or if half of the heartbeat interval has passed, the client will actively send the heartbeat.
Manual heartbeat
When dealing with time-consuming tasks, we need to make sure that the connection is made and that the heartbeat is sent actively during the task processing. So how to achieve this? Let's look at the source code of check_heartbeat().
public function check_heartbeat()
{
// ignore unless heartbeat interval is set
if ($this->heartbeat !== 0 && $this->last_read && $this->last_write) {
$t = microtime(true);
$t_read = round($t - $this->last_read);
$t_write = round($t - $this->last_write);
// server has gone away
if (($this->heartbeat * 2) < $t_read) {
$this->close();
throw new AMQPHeartbeatMissedException("Missed server heartbeat");
}
// time for client to send a heartbeat
if (($this->heartbeat / 2) < $t_write) {
$this->write_heartbeat();
}
}
}After looking at the source code, there are preconditions for sending heartbeat.
- The heartbeat interval is set
- Take the value from the socket
- Write data to socket
The first one we set manually, the third one will have last_write as long as we connect. Now we need to satisfy the second one. When will read be triggered? Of course, when will we receive the message? But we are still processing the message. Because of synchronization, we need to finish processing before receiving the next message.
Can we read it on our own initiative? Yes, we need to add a line of code to implement it, which can be invoked in consumer code.
function send_heartbeat($connection)
{
$connection->getIO()->read(0);
}We didn't get the message at this time, we just used a hack to trigger the heartbeat and see how it works.
public function read($len)
{
if (is_null($this->sock)) {
throw new AMQPSocketException(sprintf(
'Socket was null! Last SocketError was: %s',
socket_strerror(socket_last_error())
));
}
$this->check_heartbeat();
list($timeout_sec, $timeout_uSec) = MiscHelper::splitSecondsMicroseconds($this->read_timeout);
$read_start = microtime(true);
$read = 0;
$data = '';
while ($read < $len) {
$buffer = null;
$result = socket_recv($this->sock, $buffer, $len - $read, 0);
if ($result === 0) {
// From linux recv() manual:
// When a stream socket peer has performed an orderly shutdown,
// the return value will be 0 (the traditional "end-of-file" return).
// http://php.net/manual/en/function.socket-recv.php#47182
$this->close();
throw new AMQPConnectionClosedException('Broken pipe or closed connection');
}
if (empty($buffer)) {
$read_now = microtime(true);
$t_read = $read_now - $read_start;
if ($t_read > $this->read_timeout) {
throw new AMQPTimeoutException('Too many read attempts detected in SocketIO');
}
$this->select($timeout_sec, $timeout_uSec);
continue;
}
$read += mb_strlen($buffer, 'ASCII');
$data .= $buffer;
}
if (mb_strlen($data, 'ASCII') != $len) {
throw new AMQPIOException(sprintf(
'Error reading data. Received %s instead of expected %s bytes',
mb_strlen($data, 'ASCII'),
$len
));
}
$this->last_read = microtime(true);
return $data;
}After we call read, we actively trigger the heartbeat detection package, and then set last_read, which sends the heartbeat on the second manual call.
The suggestion here is to process messages as quickly as possible, and it's better not to use hack.