How can I optimize my Laravel controller to avoid 504?

Question

I'm trying to push around 3,000 items into a table - I'm not sure if its possible but I feel my controller could reach that level with some optimizations. It's currently taking 60+ seconds and timing out.

if ($request->has('items')) {
    $queueData = [];
    foreach ($request->get('items') as $item) {
        if (
            QueueItem::where('item', $item)->exists() ||
            QueueItemFailed::where('item', $item)->exists() ||
            CacheItem::where('item', $item)->exists()
        ) {
            continue;
        }

        $queueData[] = ['item' => $item];
    }

    QueueItem::insert($queueData);

    return response()->json([
        'success' => true,
    ], 200);
}

Answer 1

You have some options. The first thing is you don't need to run these queries in a loop. You know all items you are looking for so you can query for all the items on the 3 tables together at the same time (UNION of the 3 separate queries). Then you can get the difference between the original items list and the list of which of them exist in the database:

if ($request->has('items')) {
    $items = (array) $request->input('items');

    $exists = QueueItem::whereIn('item', $items)
        ->select('item')
        ->union(QueueItemFailed::whereIn('item', $items)->select('item'))
        ->union(CacheItem::whereIn('item', $items)->select('item'))
        ->pluck('item');

    // get the difference between the input items and the existing items
    $insert = collect($items)->diff($exists)->transform(function ($item) {
        return ['item' => $item];
    })->toArray();

    QueueItem::insert($insert);

    return response()->json([
        'success' => true,
    ], 200);
}

Laravel 6.x Docs - Query Builder - Unions union

Laravel 6.x Docs - Query Builder - Retrieving Results - Retrieving A List Of Column Values pluck

Laravel 6.x Docs - Collections - Available Methods - diff

Laravel 6.x Docs - Collections - Available Methods - transform

If you were to run into some parameter limit or anything like that you can chunk the $items and also chunk the $inserts the same for inserting.

Laravel 6.x Docs - Collections - Available Methods - chunk