Poor query performance with whereHas()

Question

I've been battling with this problem now for a couple of weeks.

I have a Laravel 7 application that seems to be absolutely hammering the database when performing a certain query. First I'll outline the problem and then dive a bit more into what I've tried to isolate it.

I have an Opportunity model that has a scopeLucrative() scope on the model that filters the opportunities to only show ones between the users defined threshold as per below:

public function scopeLucrative($query)
    {

        $user = auth()->user();
        $threshold = $user->preference('quality.threshold');
        $expiredThreshold = $user->preference('quality.expired_threshold');
        $hideOwnReports = $user->preference('quality.hide_own_price_changed_reports');

        /**
         * For an Opportunity to be an lucrative one, the value has to be over the user's threshold.
         */

        // Where probability is over the user's threshold
        return $query->where('probability', '>=', $threshold)

            // And where the number of false reports is less than the user's expired threshold
            ->whereHas('verifiedPriceReports', function ($report) {

                $report->where('correct_price', false)->distinct('user_id')->take(15);
            }, '<', $expiredThreshold)

            // And when the user has 'hide_own_price_changed_reports' on, hide ones they've marked as incorrect
            ->when($hideOwnReports, function ($query) use ($user) {

                return $query->whereDoesntHave('verifiedPriceReports', function ($report) use ($user) {

                    $report->where('user_id', $user->id)->where('correct_price', false);
                });
            });
    }

When called like Opportunity::with('history', 'verifiedPriceReports')->lucrative()->orderByDesc('updated_at')->paginate(10)) the database seems to be fetching a large number of rows (and takes 600ms) according to the DigitalOcean control panel despite the query only returning 10 rows as expected due to the pagination.

As you can imagine, this doesn't scale well. With only 5 active users the database queries start taking seconds to return. The query that is performed by that Query Builder is:

SELECT *
FROM `opportunities`
WHERE `probability` >= '-15'
  and (SELECT distinct count(*)
       FROM `opportunity_reports`
       WHERE `opportunities`.`id` = `opportunity_reports`.`opportunity_id`
         and `correct_price` = '0'
         and `updated_at` >= '2020-09-06 04:20:17') < 3
  and not exists(SELECT *
                 FROM `opportunity_reports`
                 WHERE `opportunities`.`id` = `opportunity_reports`.`opportunity_id`
                   and `user_id` = '1'
                   and `correct_price` = '0'
                   and `updated_at` >= '2020-09-06 04:20:17')
ORDER BY `probability` DESC
LIMIT 10 offset 0;

It didn't take long to narrow the problem down to scopeLucrative , with a simple call to the model without the lucrative scope like Opportunity::with('history', 'verifiedPriceReports')->orderByDesc('updated_at')->paginate(10)) performing as expected.

I'm at a loose end as to what I can do to fix this. Has anyone experienced anything like this before?

Answer 1

I solved this by replacing whereHas() with whereRaw()

->whereRaw('opportunities.id not in (SELECT opportunity_id
                  FROM  opportunity_reports
                    WHERE correct_price = false
                  GROUP BY opportunity_id
                  HAVING COUNT(*) >= '.$expiredThreshold.'
              )')