Output streaming with PHP ob_start & ob_get_clean

Question

I have a script that echo out content in a php script and resulting in a very large file, eg 100MB

Currently I use the following way to capture the output and write to another file

ob_start();
require_once 'dynamic_data.php'; // echo 100MB data
$data = ob_get_clean();
file_put_contents($path, $data);

Are there any easy way to re-write the above program (better not touching dynamic_data.php as it is hard to re-factor) so it can stream the output to the file directly without saving the content in the memory?

Answer 1

The ob_start documentation provides a workaround for this. You need to pass in a $output_callback and a $chunk_size .

Say you set $chunk_size to 1MB. Then every 1MB of buffered output data, your $output_callback will be called with this data and you can flush it to disk (meanwhile the output buffer is implicitly flushed).

$output_callback = function($data) {
   //$buffer contains our 1MB of output

   file_put_contents($path, $data);

   //return new string buffer
   return "";
}

//call $output_callback every 1MB of output buffered.
ob_start($output_callback, 1048576);

require_once 'dynamic_data.php';

//call ob_clean at the end to get any remaining bytes 
//(implicitly calls $output_callback final time)
ob_clean();

Answer 2

You can use proc_open and invoke the PHP interpreter with this file as argument. This will not store the data in memory but it'll create another process.

$descriptorspec = array(
   0 => array("pipe", "r"),  // stdin is a pipe that the child will read from
   1 => array("file", $path, "w"),  // stdout is a pipe that the child will write to
   2 => array("file", $path, "a") // stderr is a file to write to
);

$process = proc_open('php dynamic_data.php', $descriptorspec, $pipes);

if (is_resource($process)) {
    // $pipes now looks like this:
    // 0 => writeable handle connected to child stdin
    // 1 => readable handle connected to child stdout
    // Any error output will be appended to /tmp/error-output.txt

    fclose($pipes[0]);
    fclose($pipes[1]);
    $return_value = proc_close($process);
}
?>