从MySQL中提取大量数据

Question

Trying to pull huge data [about 200,000 rows] from MySQL using PHP. Getting the error: PHP Fatal error: Allowed memory size of xxxxx bytes exhausted

I have an application which generates almost 80-90 thousand trasnactions every month. This is a growing application, so I can expect the numbers to grow rapidly. While generating the monthly or quaterly report, I am getting the memory size error. I tried to increase the allowed memory and it worked. But I know allowing more memory is a temprary fix and is not a good practice in long run.

I need some help in scaling this application.

In my opinion, breaking the query result into multiple manageable chunks and join them together should solve the problem.

Is there any alternative/better solution?

Does NoSQL helps here? how?

Edit:

Here is the way data is used: The transaction table containes all transaction details along with the references to master data like user-id, place/location id, type of transacation.

The report will:

• Pull the records from transaction table
• Replace master references appropriately [replace user-id with name, address, etc]
• Dump the data in csv/xls format
• Store/download the file

Thanks

Answer 1

Could you not use Mysql Aggregate functions? They should always be quicker than to pull out all records and then process them. https://dev.mysql.com/doc/refman/8.0/en/group-by-functions-and-modifiers.html

Answer 2

Insert this line of code into your script at the top:

ini_set("memory_limit","128M"); // size based on your requirement if you need more than 128M then u can add this value in the place of 128M

Or another option is use Mysql command line if possible:

Use yourdatabasename
source  mysqlfile

Answer 3

A generator allows you to write code that uses foreach to iterate over a set of data without needing to build an array in memory, which may cause you to exceed a memory limit, or require a considerable amount of processing time to generate. Instead, you can write a generator function, which is the same as a normal function, except that instead of returning once, a generator can yield as many times as it needs to in order to provide the values to be iterated over. for example

function getRecords()
{
  $sql = 'SELECT field1, field2,field3 FROM table';
  $results=$db->query($sql);
  while ($row = $results->fetch_assoc())) {
    yield $row;
 }
}

foreach ($this->getRecords() as $record) {
  //process records here
}