How to run multiple sql queries using php without giving load on mysql server?

Question

I have a script that reads an excel sheet containing list of products. These are almost 10000 products. The script reads these products & compares them with the products inside mysql database, & checks

• if the product is not available, then ADD IT (so I have put insert query for that)

• if the product is already available, then UPDATE IT (so I have put update query for that)

Now the problem is, it creates a very heavy load on mysql server & it shows a message as "mysql server gone away..".

I want to know is there a better method to do this excel sheet work without making load on mysql server?

Answer 1

I am not sure if this is the case, but judging from your post, I assume it could be the case that for every check you initilize a new connection to the MySQL server. If that indeed is the case you can simply connect once before you do this check, and run all future queries trought this connection.

Next to that a good optimization option would be to introduce indexes in MySQL that would significantly speed up product search, introduce index for those product table columns, that you reference most in your php search function.

Next to that you could increase MySQL buffer size to something above 256 MB in order to cache most of the results, and also use InnoDB so you do not need to lock whole table every time you do the check, and also the input function.

Answer 2

I'm not sure why PHP has come into the mix. Excel can connect directly to a MySql database and you should be able to do a WHERE NOT IN query to add items and a UPDATE statements of ons that have changed Using excel VBA.

http://helpdeskgeek.com/office-tips/excel-to-mysql/

Answer 3

Ok, here is quick thought

Instead of running the query, after every check, where its present or not, add on to your sql as long as you reach the end and then finally execute it.

Example

$query = ""; //creat a query container
if($present) {
    $query .= "UPDATE ....;"; //Remember the delimeter ";" symbol
} else {
    $query .= "INSERT ....;";
}
//Now, finally run it
$result = mysql_query($query);

Now, you make one query at the last part.

---

Update: Approach this the another way

Use the query to handle it.

INSERT INTO table (a,b,c) VALUES (1,2,3)
  ON DUPLICATE KEY UPDATE c=c+1;

UPDATE table SET c=c+1 WHERE a=1;

Reference

Answer 4

You could try and condense your code somewhat (you might have already done this though) but if you think it can be whittled down more, post it and we can have a look.

Cache data you know exists already, so if a products variables don't change regularly you might not need to check them so often. You can cache the data for quick retrieval/changes later (see Memcached, other caching alternatives are available). You could end up reducing your work load dramatically.

Have you seperated your mysql server? Try running the product checks on a different sub-system, and merge the databases to your main, hourly or daily or whatever.