如何保存内存负载并防止ZEND HEAP php错误查询创建.txt文件的数百万条记录

Question

经过大约45分钟的ZEND HEAP错误处理记录后，PHP停顿了下来，我花了无数时间研究和尝试解决该问题，但未能做到。 在研究和尝试所有操作时，没有.ini配置可解决此问题！ 它似乎完全是php 5xx的限制或错误。

我正在查询6个表和超过54,000,000条记录。 从这些查询中，我生成一个专有文本文件。

我正在下面的脚本中寻找最佳建议，或者在重新编写脚本时都在寻找最佳建议，因为我在过去一年中对其进行了100倍的修改，并且速度越来越快，但我无法摆脱ZEND HEAD错误，该错误导致我的脚本在大量记录中被杀死集。

这不是基于Web的脚本，所有工作都已本地化并通过php cli运行。

我还应该注意，我花了无数小时来重新配置php.ini内存设置，以及我可能尝试过的其他任何可能的调整。

我希望脚本可以以OOP结构编写，从而提高内存效率。

预先感谢您的任何时间或想法。 下面是我现有的代码：

<?php
ini_set('mysql.connect_timeout', '9999999999999999999999999999999999999999999999');
ini_set('default_socket_timeout', '9999999999999999999999999999999999999999999999');
ini_set('memory_limit','9999999999999999999999999999999999999999999999M');
// Opens a connection to a MySQL server.
$connection = mysql_connect ("localhost", "root", "");
if (!$connection) 
{
die('Not connected : ' . mysql_error());
}
// Sets the active MySQL database.
$db_selected = mysql_select_db("charities", $connection);
if (!$db_selected) 
{
   die ('Can\'t use db : ' . mysql_error());
}
// Basic settings
$txtdate = date('Ymdhis');
// Filter only
$_POST["state_abbr"] = 'NY'; // State 
$_POST["loc_type"] = 'Non Profit'; // Church, School, Non Profit etc...
// Query parameters - eg for Open or Closed Status.
$_POST["case_disp"] = 'C';
$_POST["case_disp_txt"] = 'CLOSED';
$_POST["case_disp_gp"] = 'CLOSED';
$_POST["last_act_txt"] = 'CLOSED'; 
// Text output only.
$_POST["status_word"] = 'Closed'; // Location status: "ABANDONED" or "ACTIVE"
$_POST["stateU"] = 'New York'; 

$Locations_yr = "2012-" . date('Y');

// Selects all the rows in the markers table.
$query = "SELECT * FROM charity_full_merged as a 
INNER JOIN charity_full_merged_st_load_case as b
ON a.charity_id=b.charity_id
INNER JOIN charity_full_merged_land as c
ON b.charity_id=c.charity_id
INNER JOIN charity_full_merged_county as d
ON c.charity_id=d.charity_id
INNER JOIN charity_full_merged_customer as e
ON d.charity_id=e.charity_id 
WHERE
a.township_range_quarters != ''
AND 
b.geo_state = '".$_POST["state_abbr"]."' 
AND
b.casetype_txt LIKE \"%".$_POST["loc_type"]."%\"    
AND 
b.case_disp_txt = '".$_POST["case_disp_txt"]."' GROUP BY b.charity_nm;";

$result = mysql_query($query);

if($result === FALSE) {
    die(mysql_error());
}
$results_count = mysql_num_rows($result);

 $result = mysql_query($query);
     if (!$result) 
 {
      die('Invalid query: ' . mysql_error());
 }
// Creates an array of strings to hold the lines of the txt file.
$txt = array('<?xml version="1.0" encoding="UTF-8"?>');
$txt[] = "<Document>
<name>" . date('Y') . " Data Map of " . $_POST["stateU"] . ". " . $_POST["status_word"]   . " " . $_POST["loc_type"] . " Locations. </name>
<description>" . $_POST["stateU"] . " " . $_POST["status_word"]  . " " .    $_POST["loc_type"] . " " . " map." .  " " . $results_count . " Records. Created by Charity    Group 5." . date('Y') . ". </description>";
while ($row = @ mysql_fetch_assoc($result))
{
        if ($row['last_action_txt'] == 'NOP' 

        {
            $boxclr = 'activeLoc';
        }
        else if ($row['case_disp'] == 'C' 

    {
            $boxclr = 'closedLoc';
        }
    else { // Unknown
        $boxclr = "yellowBox";
}
  $txt[] = "... general content written here ... (about 100 lines of text per  record ";
} 
// End XML file
$txt[] = ' </Document>';
$txt[] = '</txt>';
$txtOutput = join("\n", $txt);

// Create .txt file.
$txtfile =  $_POST["stateU"] . "Charities" . $_POST["status_word"] . "-" .     $_POST["loc_type"] . "-LocationS-" . $results_count . "-" . $txtdate . ".txt";

// Put the contents of $txtOutput into the $txtfile.
file_put_contents($txtfile, $txtOutput);

echo "$results_count " . $_POST["status_word"] . " " . $_POST["loc_type"] . " Location     records processed...";
?>

Answer 1

我认为您已经了解了一般性问题。 您正在尝试将整个结果集累积到$txt数组中-如您所见，这不适用于庞大的数据集。 相反，您应该允许PHP在生成数据时输出数据。

所以代替：

$txt[] = "(header stuff)";
while ($row = mysql_fetch_assoc($result) {  // Remove the @ here!!!
   $txt[] = "(content stuff)";              // This becomes 'huge'
}
$txt[] = "(footer stuff)";

$txtOutput = join("\n", $txt);              // Now your memory usage is
                                            // ~ 2 * 'huge'

file_put_contents($filename, $txtOutput);

你应该fopen提前时刻的文件，并fwrite你需要什么，当您去。

$fp = fopen($filename, "w");

fwrite($fp, "(header stuff)");

while ($row = mysql_fetch_assoc($result) {
   fwrite($fp, "(content stuff)");
}

fwrite($fp, "(footer stuff)");

fclose($fp);

此外，当您必须尝试使用​​诸如9999999999999999999999999999999999999999999999数字作为套接字超时或内存限制时，这应该是对您做错了一个大警告。