PHP output buffer not flushing

Question

I have a number of scripts that echo progress as they execute because they are long running. Each basically does the following at the end of each looped row of data processed:

echo '.';
@ob_flush();
flush();

This was working just fine for years and then I upgraded to PHP 5.3.x and Apache 2.2.x across several servers. Now, even if I pad the buffer with white space or set "ob_implicit_flush(1)", I can't get it to show the output on command.

One server still shows the output, but it is in chunks. It can take nearly 5 minutes and then suddenly a string of dots appears on the screen. With the other servers, I get nothing until the script finishes executing entirely.

I've tried looking through the php.ini and httpd.conf files to see if I could figure out what had changed between the different servers, but I'm obviously missing something.

I've also tried disabling mod_deflate in .htaccess for the affected scripts, but that doesn't help either (disabling mod_gzip used to fix the problem right away).

Can someone point me in the right direction with this please? Not being able to monitor script execution in real time is causing all sorts of problems but we can't stay on these older PHP versions any longer.

On an even more peculiar side note, I did try downgrading a server to PHP 5.2.17 but the output buffer problem remained after the downgrade. This makes me suspect it is something relating to the way Apache is handling PHP output since Apache 2 was left in place.

Answer 1

ob_flush() (an flush()) only flush the PHP buffer - the webserver maintains a buffer itself. And weird as it may sound, flushing the buffer early actually decreases throughput on the server, hence recent versions of apache buffer more agressively. There's also horrendous problems relating to compression and partial rendering when working with HTTP chunked encoding.

If you want to incrementally add content to a page then use ajax or websockets to add it a bit at a time.

Answer 2

This issue has more to do with your server (apache) rather than the php version.

One option is to disable output buffering, though performance might suffer in other parts of the site

On Apache

Set the php ini directive ( output_buffering=off ) from your server config, including a .htaccess file. So I used the following in a .htaccess file to disable the output_buffering just for that one file:

<Files "q.php">
    php_value output_buffering Off
</Files>

And then in my static server configuration, I just needed AllowOverride Options=php_value (or a larger hammer, like AllowOverride All ) in order for that to be allowed in a .htaccess file.

On Nginx

To Disable buffering for Nginx (add "proxy_buffering off;" to the config file and restart Nginx

Answer 3

Most probably the change as described in the original question is that the new setup was using FastCgi ( http://www.fastcgi.com/mod_fastcgi/docs/mod_fastcgi.html ) and this has buffering turned on be default.

But there are other factors to check:

If you are using Fcgid then this also has buffering: http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#fcgidoutputbuffersize

If your charsets between PHP and Apache don't match - this can hold things up.

mod_deflate and mod_gzip also buffer (as mentioned in original question)

So steps to check are:

1. Flush the PHP buffer - (as described in the question)

2. Turn off the Apache buffering - Add php_value output_buffering off in .htaccess

3. Disable mods that buffer for deflation - Disable mod_deflate and mod_gzip

4. Ensure your char encoding matches between PHP and Apache - add default_charset = "utf-8"; in php.ini and AddDefaultCharset utf-8 in httpd.conf)

5. Disable buffering in FastCgi or Fcgid - You can disable buffering in FastCgi by adding the -flush option. Details in the link above. Option for Fcgid also listed above.

As far as I know, those are the only buffers on the server; obviously other devices between server and browser may also buffer, eg a proxy may wait for the full output to be provided before passing it on. Fiddler ( https://www.telerik.com/fiddler ) does this, and it frequently floors me until I remember.

Answer 4

There is a possible work around for this issue that doesn't require you to edit your existing scripts or modify your server configuration to stop buffering output. By using a wrapper script you can start your long running process in the background from the php script serving the web request. Then you can pipe the output of the long running process into a text file which can be easily read to find the current progress of the script via polling. Example below:

Long running process script

<?php
// long_process.php
echo "I am a long running process ";
for ($i = 0; $i < 10; $i++) {
    echo ".";
    sleep(1);
}
echo " Processing complete";
?>

Script to initialize the long running process and watch output

<?php
    // proc_watcher.php
    $output = './output.txt';
    if ($_GET['action'] == 'start') {
        echo 'starting running long process<br>';
        $handle = popen("nohup php ./long_process.php > $output &", 'r');
        pclose($handle);
    } else {
        echo 'Progress at ' . date('H:i:s') . '<br>';
        echo file_get_contents($output);
    }
    $url = 'proc_watcher.php';
?>
<script>
    window.setTimeout(function() {
         window.location = '<?php echo $url;?>';
    }, 1000);
</script>

If you send a web request to proc_watcher.php?action=start the script should start the long running process in the background and then return the contents of the output file to the web browser every second.

The trick here is the command line nohup php ./long_process.php > ./output.txt & , which runs the process in the background and sends the output to a file instead of STDOUT.

Answer 5

I tried everything to get this to work, including all known settings listed above. I've been trying to use PHP to serve chunked video files using HTTP_RANGE and it wasn't working.

After pulling most of my hair out, i found the answer: you have to output at least one byte more than the buffer size in order for it to output to the browser. Here's the script that ended up working for me:

<?php 
// Close open sessions
session_write_close();

// Turn off apache-level compression
@apache_setenv('no-gzip', 1);

// Turn off compression
@ini_set('zlib.output_compression', 0);

// Turn error reporting off
@ini_set('error_reporting', E_ALL & ~ E_NOTICE);

// Tell browser not to cache this
header("Cache-Control: no-cache, must-revalidate");

// close any existing buffers
while (ob_get_level()) ob_end_clean();

// Set this to whatever you like
$buffer = 8096;

for($i = 1; $i <= 8; $i++) 
{
    // Start a output buffer with specified size
    ob_start(null,$buffer,PHP_OUTPUT_HANDLER_FLUSHABLE);
    // Output exactly one byte more than that size 
    // \n == 2 bytes, so 8096-1+2 = 8097
    echo str_repeat('=', $buffer-1)."\n";
    // 0.25s nap
    usleep(250000);
    // End output buffering and flush it
    ob_end_flush();
    flush();
}

Hope this helps someone!