如何保存内存负载并防止ZEND HEAP php错误查询创建.txt文件的数百万条记录

Question

PHP chokes up after about 45 minutes of processing records with a ZEND HEAP error which I spent countless hours researching and trying to resolve and was unable to do so. There is not an .ini configuration that resolves this as I researched and tried everything! It solely appears to be a php 5xx limitation or bug.

I have 6 tables and over 54,000,000 records I am querying. From these queries I generate a proprietary text file.

I am looking for the best recommendations in my below script or recreation of the script as I've modified it 100x times over the past year and it's gotten faster but I cannot get away from the ZEND HEAD error which kills my script on very large record sets.

This is not web based script all work is localized and run through the php cli.

I should also note I spent countless hours re configuring php.ini memory settings and any other possible tweak I could try with no luck.

I am hoping the script can be written in an OOP structure that allows much more memory efficiency.

Thanks in advance for any times or ideas. Below is my existing code:

<?php
ini_set('mysql.connect_timeout', '9999999999999999999999999999999999999999999999');
ini_set('default_socket_timeout', '9999999999999999999999999999999999999999999999');
ini_set('memory_limit','9999999999999999999999999999999999999999999999M');
// Opens a connection to a MySQL server.
$connection = mysql_connect ("localhost", "root", "");
if (!$connection) 
{
die('Not connected : ' . mysql_error());
}
// Sets the active MySQL database.
$db_selected = mysql_select_db("charities", $connection);
if (!$db_selected) 
{
   die ('Can\'t use db : ' . mysql_error());
}
// Basic settings
$txtdate = date('Ymdhis');
// Filter only
$_POST["state_abbr"] = 'NY'; // State 
$_POST["loc_type"] = 'Non Profit'; // Church, School, Non Profit etc...
// Query parameters - eg for Open or Closed Status.
$_POST["case_disp"] = 'C';
$_POST["case_disp_txt"] = 'CLOSED';
$_POST["case_disp_gp"] = 'CLOSED';
$_POST["last_act_txt"] = 'CLOSED'; 
// Text output only.
$_POST["status_word"] = 'Closed'; // Location status: "ABANDONED" or "ACTIVE"
$_POST["stateU"] = 'New York'; 

$Locations_yr = "2012-" . date('Y');

// Selects all the rows in the markers table.
$query = "SELECT * FROM charity_full_merged as a 
INNER JOIN charity_full_merged_st_load_case as b
ON a.charity_id=b.charity_id
INNER JOIN charity_full_merged_land as c
ON b.charity_id=c.charity_id
INNER JOIN charity_full_merged_county as d
ON c.charity_id=d.charity_id
INNER JOIN charity_full_merged_customer as e
ON d.charity_id=e.charity_id 
WHERE
a.township_range_quarters != ''
AND 
b.geo_state = '".$_POST["state_abbr"]."' 
AND
b.casetype_txt LIKE \"%".$_POST["loc_type"]."%\"    
AND 
b.case_disp_txt = '".$_POST["case_disp_txt"]."' GROUP BY b.charity_nm;";

$result = mysql_query($query);

if($result === FALSE) {
    die(mysql_error());
}
$results_count = mysql_num_rows($result);

 $result = mysql_query($query);
     if (!$result) 
 {
      die('Invalid query: ' . mysql_error());
 }
// Creates an array of strings to hold the lines of the txt file.
$txt = array('<?xml version="1.0" encoding="UTF-8"?>');
$txt[] = "<Document>
<name>" . date('Y') . " Data Map of " . $_POST["stateU"] . ". " . $_POST["status_word"]   . " " . $_POST["loc_type"] . " Locations. </name>
<description>" . $_POST["stateU"] . " " . $_POST["status_word"]  . " " .    $_POST["loc_type"] . " " . " map." .  " " . $results_count . " Records. Created by Charity    Group 5." . date('Y') . ". </description>";
while ($row = @ mysql_fetch_assoc($result))
{
        if ($row['last_action_txt'] == 'NOP' 

        {
            $boxclr = 'activeLoc';
        }
        else if ($row['case_disp'] == 'C' 

    {
            $boxclr = 'closedLoc';
        }
    else { // Unknown
        $boxclr = "yellowBox";
}
  $txt[] = "... general content written here ... (about 100 lines of text per  record ";
} 
// End XML file
$txt[] = ' </Document>';
$txt[] = '</txt>';
$txtOutput = join("\n", $txt);

// Create .txt file.
$txtfile =  $_POST["stateU"] . "Charities" . $_POST["status_word"] . "-" .     $_POST["loc_type"] . "-LocationS-" . $results_count . "-" . $txtdate . ".txt";

// Put the contents of $txtOutput into the $txtfile.
file_put_contents($txtfile, $txtOutput);

echo "$results_count " . $_POST["status_word"] . " " . $_POST["loc_type"] . " Location     records processed...";
?>

Answer 1

I think you already understand the general problem. You're trying to accumulate the entire result set into the $txt array - that's not going to work for huge data sets, as you're seeing. Instead, you should allow PHP to output the data as it is generated.

So instead of:

$txt[] = "(header stuff)";
while ($row = mysql_fetch_assoc($result) {  // Remove the @ here!!!
   $txt[] = "(content stuff)";              // This becomes 'huge'
}
$txt[] = "(footer stuff)";

$txtOutput = join("\n", $txt);              // Now your memory usage is
                                            // ~ 2 * 'huge'

file_put_contents($filename, $txtOutput);

You should fopen the file ahead of time, and fwrite what you need as you go.

$fp = fopen($filename, "w");

fwrite($fp, "(header stuff)");

while ($row = mysql_fetch_assoc($result) {
   fwrite($fp, "(content stuff)");
}

fwrite($fp, "(footer stuff)");

fclose($fp);

Additionally, when you have to try and use numbers like 9999999999999999999999999999999999999999999999 for a socket timeout or a memory limit, that should be a big warning that you're doing something wrong.