PHP While Loop Slowdown over time

Question

I have seemingly harmless while loop that goes through the result-set of a mysql query and compares the id returned from mysql, to one in a very large multidimensional array:

//mysqli query here
while($row = fetch_assoc())
{
    if(!in_array($row['id'], $multiDArray['dimensionOne']))
    {
        //do something
    }
}

When the script first executes, it is running through the results at about 2-5k per second. Sometimes more, rarely less. The result set brings back 7million rows, and the script peaks at 2.8GB of memory.

In terms of big data, this is not a lot.

The problem is, around the 600k mark, the loop starts to slow down, and by 800k, it is processing a few records a second.

In terms of server load and memory use, there are no issues.

This is behaviour I have noticed before in other scripts dealing with large data sets.

Is array seek time progressively slower as the internal pointer moves deeper?

Answer 1

That really depends on what happens inside the loop. I know you are convinced it's not a memory issue but it looks like one. Program usually get very slow when system tries to get extra RAM by using SWAP. Using hard drive is obviously very slow and that's what you might be experiencing. It's very easy to benchmark it.

In one terminal run

vmstat 3 100

Run you scrip and observe vmstat. Look into IO and SWAP. If that is really not the case then profile execution with XDEBUG. It might be tricky because you do many iterations and this will also cause major IO.