PHP loop - poor SQL performance

Question

I'm looping through 40 associative arrays:

array(
  'key0' => value,
  'url0' => value,
  'tit0' => value,
  'cdn0' => value,
  'cdn1' => value,
  'cdn2' => value,
)

and I am performing multiple select and one possible insert queries. I tried to optimize the performance by reducing the amount of queries.

foreach($buf){
  $sth1->execute();//SELECT * FROM metadata WHERE url = '{$buf['url0']}'
  $sth2->execute();//SELECT * FROM metadata WHERE key = '{$buf['key0']}'

  if(!$ret=$sth1->fetch(PDO::FETCH_ASSOC)){
    if($sth2->fetch(PDO::FETCH_ASSOC)){
      die( 'key duplicate - error' );
    }

    $data[ ] = $buf;

    $sth3->execute();//INSERT INTO metadata ...
  } else {
    $data[ ] = $ret; 
}

This is however slow (the loop takes about 4.2 seconds). I tried making it faster by removing a query.

foreach($buf){
  $sth1->execute();//SELECT * FROM metadata WHERE url = '{$buf['url0']}' OR key = '{$buf['key0']}'

  if(!$ret=$sth1->fetch(PDO::FETCH_ASSOC)){       
    $sth3->execute();//INSERT INTO metadata ...

    $data[ ] = $buf;
  } else {
    if($ret['key']==$generated_key){die('key duplicate - error');}

    $data[ ] = $ret; 
}

This for some reason made it even slower (5-6s). Thus, I'm left clueless. How can I make this have a reasonable load time? I tried putting $sth2->execute in the if(!$ret...) statement but that doesn't give me any speed gain either.

It doesn't seem to be the INSERT that is the problem due to most of the array data being already IN the database. Whenever I run queries in phpmyAdmin it does it in 0.0000000003 seconds so it must have something to do with the loop.

Answer 1

Speedup #1: Move $sth2->execute(); to right after the first if . You don't seem to need the result if the url test succeeds.

Speedup #2: Be sure to have INDEX(url) and INDEX(key) ;

Speedup #3: Change INDEX(key) to UNIQUE(key) and skip the SELECT ... key ; simply check for dup key after doing the INSERT .

Speedup #4: (This one may or may not help.) Do all of the SELECT ... url in a single query: SELECT ... url IN (40-urls-in-list) . (Requires Speedup #2.) Save the results in an associative array and walk through it to do the rest of your SELECT/INSERT stuff.

Speedup #5: Build a 'batch' INSERT (multiple rows in a single INSERT ), then execute it at the end of the 40-item loop.

Please provide SHOW CREATE TABLE .

MyISAM? or InnoDB? Have you tuned innodb_buffer_pool_size to be about 70% of available RAM? If not, this might improve performance.

Answer 2

Is what you considered that possibility? https://dev.mysql.com/doc/refman/5.0/en/insert-select.html

Answer 3

Unfortunately, there is not enough information about your MySQL indexing strategy or amount of information you select.

Possible issues that come to my mind in first second:

• when you perform queries from MySQL you hit query cache, from loop cache is at some point expired
• poor indexing strategy in database - selection by varchar is not always best solution, using char(16) and MD5 sum (or something similar) is probably is a bit better approach
• you have some unnoticed loop in iterations

Solution to all those would be to echo a lot of debug data with precision timestamp, read them over and log SQL queries to MySQL server log and run all these in one batch in console/PHPMyAdmin for better debugging.