PHP / PDO-以高性能方式将大表输出为JSON

Question

I've some queries which will output large Tables. Reduced the code to the most important thing, it looks like:

$pdo = new PDO(...);
$statement = $pdo->prepare($query);
$statement->execute($data);

$largeArray = $statement->fetchAll(PDO::FETCH_ASSOC);
$largeString = json_encode($largeArray);
echo $largeString;

It's working fine. But what If I want to produce some realy large outputs? The above code, has to store the full output in the memory.

One alternative would be:

$pdo = new PDO(...);
$statement = $pdo->prepare($query);
$statement->execute($data);

echo '[';
if($line = $statement->fetch(PDO::FETCH_ASSOC)) {
  echo json_encode($line);
  while($line = $statement->fetch(PDO::FETCH_ASSOC)) {
    echo ',';
    echo json_encode($line); 
  }
}
echo ']';

But does this run with good performance? What would be more performant solutions?

Answer 1

What I would do is fetch in batches of lets say 200

Why : Every time you load a batch, you are loading into the machine memory which is of course pretty limited.

I suggest saving the results into files of 2000

(the batches or file sizes may vary, and you should probably find the sweet spot for your system but keep it small)

Why : Even though this is not a must, specially if you are going to load this in batches next time you need it. But if you are planing on opening it in any editor you will surely need to wait a while or even cause the app to crash.

Also might I suggest saving the data as csv if possible instead of json, csv has a much smaller footprint for huge dumps. Also when loading from a json file you will not be able to load data in batches without doing some hacking around. with csv you can simply go line by line.