如何使用PHP处理大量数据？

Question

My web application lets user import an excel file and writes the data from the file into the mysql database. The problem is, when the excel file has lots of entries, even 1000 rows, i get an error saying PHP ran out of memory. This occurs while reading the file. I have assigned 1024MB to PHP in the php.ini file.

My question is, how to go about importing such large data in PHP.

I am using CodeIgniter. for reading the excel file, i am using this library.

---

SOLVED. I used CSV instead of xls. and I could import 10,000 rows of data within seconds. Thank you all for your help.

Answer 1

As others have said, 1000 records is not much. Make sure you process the records one at a time, or a few at a time, and that the variables you use for each iteration go out of scope after you're finished with that row or you're reusing the variables.

If you can avoid the necessity of processing excel files by exporting them to csv, that's even greater, cause then you wouldn't need such a library (which might or might not have its own memory issues).

Don't be afraid of increasing memory usage if you need to and that solves the problem, buying memory is the cheapest option sometimes. And don't let the 1 GB scare you, it is a lot for such a simple task, but if you have the memory and that's all you need to do, then its good enough for the moment.

And as a plus, if you are using an old version of PHP, try updating to PHP 5.4 which handles memory much better than its predecessors.

Answer 2

Instead of inserting one a time in a loop. Insert 100 row at a time.

You can always run

INSERT INTO myTable (clo1, col2, col2) VALUES 
(val1, val2), (val3, val4), (val5, val6) ......

This way number of network transaction will reduce thus reducing resource usage.