Converting multiple sql queries to one large array

Question

My first question, I hope I'll do it ok.

I'm working in a web engine in PHP that another person wrote. This code makes a lot of database calls. For instance, to write a html tag, it makes three calls to database: one for asking the filename, another one for the title, and another one for the alt attribute, even when title and alt are in the same database field.

To improve this code, I'm trying to make one initial call that brings all the info I'll need after and saves it into an array. As my initial call uses several joins, I'm using a bidimensional array, for better understanding:

SELECT a.*, b.*, c* FROM a INNER JOIN b ON... ... INNER JOIN c ON...  

I save the results in a bidimensional array:

$info["a"]["field1"], $info["a"]["field2"], ...  
$info["b"]["field1"], $info["b"]["field2"], ...  
$info["c"]["field1"], $info["c"]["field2"], ...  

So each time I need data, I search in this array instead of making a database call.

My question is: given that I start with a big multitable query and then I use a big bidimensional array, am I really improving this code in terms of performance?

Thanks.

Answer 1

I'd say that depends on you infrastructure.

If you are running MySQL on the same host as the application, it might not improve performance, or even hurt it.

Just test it in your exact environment.

I wrote an article about reducing network latencies. Although Oracle specific, some methods will also apply to MySQL: http://blog.fatalmind.com/2009/12/22/latency-security-vs-performance/

Answer 2

I'd say you most probably shouldn't touch it. For two reasons.

1. Database statements are fast. That is what a database is all about. Using a unique index, a SELECT from a database table reads a very limited amount of data from the hard disk (or, maybe even the database cache in memory). Depending on the access to the database (local, remote) it should either be extremely fast or lightning fast. Unless you have a bad network.

2. You add code to the project. Added code means the possibility of bugs. While the "old" routine does it's job fine and you have no obvious performance problem (at least you do not state that you want to change this because your performance is low), your code will be new, untested and will most likely contain bugs and/or an badly performing search through this array. Add to that the fact that an in-memory-search over unindexed data like arrays is much slower than seeking in indexed data and you will most probably hurt the performance.

Answer 3

Making joined queries in a big database takes significant amount of query time. Creating a giant array wouldn't be the solution to that becouse declaring giant array and using only a few entries in each run is redundant.

What you are looking for is memcache. Memcache works like an array on the server's ram. Php do not declare the array and cache it every time. Instead, memcache keeps the data in the ram ready-to-use. You can design your id-based definition in memcache and use it.

My suggestion would be declaring an object extending the array class. You can overload the constructor to retrieve the data if it's in the cache. If not, it should get the data from database and add to memcache.

The data will be in the cache as long as you defined.

Try not to make giant arrays.

http://php.net/manual/en/book.memcache.php

This is the class that i've been using. If you have tables that consists of constant data i suggest you to use cacheTable method with 88000 sec time limit and add a cronjob to re-cache it daily.

<?php
    class mc extends ArrayObject{
        var $mc_obj;
        function __construct(){
            $this->mc_obj = new Memcache;
            $this->mc_obj->connect("127.0.0.1",11211);
            # You might need to set "127.0.0.1" to "localhost"
        }

        function fetch($table,$id){
            /*
                if the n-th row of the $table is not in the cache, cache it
                return the row.
            */
            if(!$this->mc_obj->get($table."_".$id)){
                 $this->cacheById($table,$id);
            }
            return $this->mc_obj->get($table."_".$id);

        }
            /*
                numeric is the boolean for mysql_fetch type.
                if true the the array will be created numeric with mysql_fetch_row
                else the array will be created associative with mysql_fetch_assoc
            */
        function cacheTable($table,$numeric=false,$conditions="1",$idFieldOfTable="id",$cacheTimeLimit=120){
            $q1 = mysql_query("select * from `".$table."` where ".$conditions.";");
            if($numeric){
                while($row = mysql_fetch_row($q1)){
                    $this->mc_obj->set($table."_".$row[0],$row,false,$cacheTimeLimit);
                }
            }
            else{
                while($row = mysql_fetch_assoc($q1)){
                    $this->mc_obj->set($table."_".$row[$idFieldOfTable],$row,false,$cacheTimeLimit);
                }
            }

        }

        function cacheById($table,$id,$numeric=false,$idFieldOfTable="id",$cacheTimeLimit=120){

            $q1 = mysql_query("select * from `".$table."` where `".$idFieldOfTable."`=".$id." Limit 1;");
            if($numeric){
                $row = mysql_fetch_row($q1);
            }
            else{
                $row = mysql_fetch_assoc($q1);
            }
            $this->mc_obj->set($table."_".$id,$row,false,$cacheTimeLimit);
        }
        public function offsetGet($key) {
            $id = substr($key,strrpos($key,"_")+1);
            $table = substr($key,0,strrpos($key,"_"));

            return $this->fetch($table,$id);
            /*
                By overriding this method you will be able to call any particular cell like
                $cell = $memCacheObject['tableName_Rowid']['field'];
            */
        }
    }
?>

Here is how to use this :

<?php
mysql_connect("localhost","root","");
mysql_select_db("DB_NAME");
$db = new mc();

$table = "city";
$id = 3;
$a = $db->fetch($table,$id);
// $a is the associative row array

$b = $db->fetch($table,$id);
// $b is the numeric row array

$cell = $db['city_18']['name'];
//$cell is the name of the city with id 18 in the city table

//in order to cache the whole table
$db->cacheTable("city",false,"1","id",90000);

?>