Combining Transactions WITH Queues in Laravel

Question

Is there a way to tell the Queue facade to not automatically persist what was pushed to the queue until I specifically tell it to?

Queue::noPersist()
Queue::push()
Queue::push()
Queue::persist()

See my problem is as follows, my code execution is wrapped in a transaction.

Inside this transaction is some eloquent records being added and right after also a job being pushed to the queue.

queue runs before transaction commits: The issue is that the queue sometimes starts running before my transaction finishes so it tries to reference a record that hasn't yet been " committed ".

Provided my design is ok I'm thinking what I really need is for the queue to only submit after the transaction successfully commits: If the transaction fails then the job should also be rolled back so to speak. Or in other words the jobs shouldn't be pushed until after the transaction succeeds.

queue delay I saw laravel has a delay I can add to the queue but I didn't want to hack a random number in there as that's a bit fragile I really just want to only submit the queues if transaction succeeds.

Any guidance or built in features that could be useful to help me out here?

DB::transaction(function() {
    // add record A to DB
    // add JOB to QUEUE (this job starts firing before transaction commits and fails finding the record A)
    // more operations
});

Answer 1

Commit happens after your closure function, so if you keep using this structure - you won't be able to enqueue after commit.

However, Laravel got another way to do transactions (see 'Manually Using Transactions' in Laravel documentation ). You could do something like:

$errors = false;

try {
  DB::beginTransaction();

  // All your operations here
} catch (Exception $e) {
  $errors = true;
  DB::rollBack();
}

if (!$errors) {
  DB::commit();

  // Add to queue
}

Answer 2

You can use my package: https://github.com/therezor/laravel-transactional-jobs

Simply add public $afterTransactions = true; to your job and it will be fired after transaction successfully committed.

Answer 3

I faced same problem working with rabbitmq in golang. I was able to solve this problem by implementing an idempotency system which did 2 things:

1. Check if a message received from the queue, has been already processed. (Duplicate detection)
2. Check if a message's parent context has been solved successfully. This means, the transaction that has pushed this message in the queue has been solved or not.

In your particular case, the second point should fix the problem. I used 2 keys in message headers, namely :

1. Current-Idempotency-Key : used to implement point 1, above.
2. Parent-Idempotenct-Key. : used to implement point 2, above. This is the idempotency key of the parent message due to which the transaction block got initiated. Once the transaction is done, I would commit the Parent-Idempoptenct-Key to Database.

Under the above system, when a queue push is performed in middle of a transaction block, then even if a message is received before the commit is performed in the parent context, it will still not start its processing because at the receiver:

1. We allow processing messages whose parent-key has been commited. (implemented using a simple query check in database)
2. Messages whose parent-key has not been committed, we send them in wait_queue with a configured TTL via deadlettering. This allows them to be retried again after the TTL expires, until then the transactions should have been committed.

NOTE : This system works well only when you have are fetching messages from a queue, doing some transactions on them (parent context) and then pushing them again in a queue ie a Queue based data pipeline.

Answer 4

You can implement your own __wakeup method to handle the ModelNotFoundException raised by the SerializesModels trait, using something like this:

use Queueable, SerializesModels {
    __wakeup as wakeUp;
}

/**
 * Restore the model after serialization.
 *
 * @return void
 */
public function __wakeup()
{
    try
    {
        // Call SerializesModel __wakeup
        $this->wakeUp();
    }
    catch (Illuminate\Database\Eloquent\ModelNotFoundException $e)
    {
        // Delete job from the queue
        $this->delete();
    }
}