Symfony 4 worker using doctrine not working properly : SQLSTATE[HY000] [2002] Connection timed out

Question

I'm using a worker with the Symfony 4 messenger component.

This worker is

• receiving a message (from rabbitMQ)
• launch ffmpeg
• do a treatment on a video
• and save something in a database.

To configure this worker on Symfony I've done this (middleware are important):

// config/packages/framework.yaml
framework:
    messenger:
        buses:
            command_bus:
                middleware:
                    # each time a message is handled, the Doctrine connection
                    # is "pinged" and reconnected if it's closed. Useful
                    # if your workers run for a long time and the database
                    # connection is sometimes lost
                    - doctrine_ping_connection

                    # After handling, the Doctrine connection is closed,
                    # which can free up database connections in a worker,
                    # instead of keeping them open forever
                    - doctrine_close_connection

        transports:
            ffmpeg:
              dsn: '%env(CLOUDAMQP_URL)%'
              options:
                auto_setup: false
                exchange:
                    name: amq.topic
                    type: topic
                queues:
                  ffmpeg: ~

        routing:
            # Route your messages to the transports, for now all are AMQP messages
            'App\Api\Message\AMQPvideoFFMPEG': ffmpeg
        ## Handle multiple buses ? https://symfony.com/doc/current/messenger/multiple_buses.html
        ## When queries and command should be distinguished

Then in order to understand what may cause this issue I've try to debug the messenger to see if the middleware are correctly configured

root@b9eec429cb54:/var/www/html# php bin/console debug:messenger

Messenger
=========

command_bus
-----------

 The following messages can be dispatched:

 ------------------------------------------------------ 
  App\Api\Message\AMQPvideoFFMPEG                       
      handled by App\Api\Message\Handler\FFMPEGHandler  
 ------------------------------------------------------ 

Everything seems ok right ?

So how is this possible to see this :

[2019-08-23 10:25:26] messenger.ERROR: Retrying App\\Api\\Message\\AMQPvideoFFMPEG - retry #1. {"message":"[object] (App\\Api\\Message\\AMQPvideoFFMPEG: {})","class":"App\\Api\\Message\\AMQPvideoFFMPEG","retryCount":1,"error":"[object] (Doctrine\\DBAL\\Exception\\ConnectionException(code: 0): An exception occurred in driver: SQLSTATE[HY000] [2002] Connection timed out at /var/www/html/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php:93, Doctrine\\DBAL\\Driver\\PDOException(code: 2002): SQLSTATE[HY000] [2002] Connection timed out at /var/www/html/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:31, PDOException(code: 2002): SQLSTATE[HY000] [2002] Connection timed out at /var/www/html/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:27)"} []

I'm completely lost, Have I missed something ?

This happens sometimes, but it works most of the time, I suppose this bug happen when my worker has lost the connection to DB especially if ffmpeg treatment last 7 minutes or higher, but this should be avoided by the ping and the close connection's middlewares. So i don't clearly understand what is the problem here.

Answer 1

After reading the code of my middlewares and especially this block

https://github.com/symfony/symfony/blob/4.4/src/Symfony/Bridge/Doctrine/Messenger/DoctrinePingConnectionMiddleware.php

class DoctrinePingConnectionMiddleware extends AbstractDoctrineMiddleware
{
    protected function handleForManager(EntityManagerInterface $entityManager, Envelope $envelope, StackInterface $stack): Envelope
    {
        $connection = $entityManager->getConnection();
        if (!$connection->ping()) {
            $connection->close();
            $connection->connect();
        }
        if (!$entityManager->isOpen()) {
            $this->managerRegistry->resetManager($this->entityManagerName);
        }
        return $stack->next()->handle($envelope, $stack);
    }
}

We can see that my handler is called right after the connection openning. This behaviour is supposed to work, I assume it is, but FFMPEG can work during a long time with the same RabbitMQ's message. So the last step of my handler that would insert something into the database can provide a mySQL has gone away error, or connection timed out.

That's why, I took this snippet and put it into a method without the call of the handler stuff only the code related to doctrine connect, then I call this just before any insert into my DB like this :

public function __invoke(AMQPvideoFFMPEG $message)
    {
        // reset connection if not found
        $this->processService->testConnection();
        $process = $this->processService->find($message->getProcess());
        $this->renderServcie->updateQueue($process->getQueue(), "processing");

// some other stuff
}

Where testConnection() method is

 /**
     * Reconnect if connection is aborted for some reason
     */
    public function testConnection()
    {
        $connection = $this->entityManager->getConnection();
        if (!$connection->ping()) {
            $connection->close();
            $connection->connect();
        }

    }

But I've experimented another issue after that

Resetting a non-lazy manager service is not supported. Set the "doctrine.orm.default_entity_manager" service as lazy and require "symfony/proxy-manager-bridge" in your composer.json file instead.

After installing "symfony/proxy-manager-bridge", the error was gone.

So far no connection timed out was experienced. Wait and see.

Answer 2

Simply disconnect before any insert operation:

public function handle(…)
{
    // your time-consuming business logic

    // disconnect if needed
    if (!$this->entityManager->getConnection()->ping()) {
        $this->entityManager->getConnection()->close();
    }

    // save your work
    $this->entityManager->flush();
}