[英]fetching millions of records from the database using php
我在使用 php 從 mysql 數據庫中獲取數百萬條記錄時遇到問題，我想將這些記錄加載到 excel 文件中。當我執行查詢時，如果結果集的記錄很少（數百條），那么它會保存到 excel 中文件，但是當我獲取數百萬條記錄時，它會顯示一個名為
內部服務器錯誤
服務器遇到內部錯誤或配置錯誤，無法完成您的請求。 請聯系服務器管理員，告知錯誤發生的時間以及您可能所做的任何可能導致錯誤的事情。
服務器錯誤日志中可能提供有關此錯誤的更多信息。
我的代碼是：
<?php
/*
author: shobhan
Date: 20-10-2014
Description: This file creates excel sheet from the database based on the selection and available for download
*/
ini_set('max_execution_time', 6000);
ini_set('memory_limit','1000M');
set_time_limit(0);
//echo ini_get('memory_limit');
include("db/config.php"); //include global configuration file
global $db_host,$db_name,$db_user,$db_password; //include global variables
$con=mysqli_connect($db_host,$db_user,$db_password,$db_name);
$flag=1;
$download_filename="";
header("Content-type: text/csv");
header("Content-Disposition: attachment; filename=suburban_data.csv");
// Disable caching
header("Cache-Control: no-cache, no-store, must-revalidate"); // HTTP 1.1
header("Pragma: no-cache"); // HTTP 1.0
header("Expires: 0"); // Proxies
//echo $query;
//echo "<br/><br/><br/><br/><br/><br/><br/><br/>";
$result_array=array();
$output = fopen("php://output", "w");
$headings=array("Patient Name","Mobile","Visit Date","Centre","Profiles","Gender","Age");
fputcsv($output, $headings);
function preparewhere($ptstring){
global $db_host,$db_name,$db_user,$db_password; //include global variables
global $download_filename;
global $flag;
$year=$_POST['year'];
$centre=$_POST['centre_ids'];
$gender=$_POST['gender'];
$profile=$_POST['profile_ids'];
$testresult=$_POST['testresult'];
$age=$_POST['age'];
$treatmenttype=$_POST['treatmenttype'];
$where="";
$year_condition="";
$centre_condition="";
$gender_condition="";
$test_condition="";
$testresult_condition="";
$age_condition="";
if($year=="0"){
$year_condition="Visit.VstDate >= '2013-01-01'";
}
else if($year=="2013"){
$year_condition="Visit.VstDate >= '2013-01-01' and Visit.VstDate < '2014-01-01'";
$download_filename.="2013";
}
else{
$year_condition="Visit.VstDate >= '2014-01-01'";
$download_filename.="2014";
}
//centre condition
if($centre!=0){
$centre_condition=" and centres.SysNo in(".$centre.")";
$download_filename.="_".$centre;
}
//gender condition
if($gender!="0"){
$gender_condition=" and patient.PatSex= '".$gender."'";
$download_filename.="_".$gender;
}
if($flag==2){ //if input contains only profiles
$test_condition=" and VisitProfile.ProfCode in(".$ptstring.")";
}
else{
$test_condition=" and Test.TestCode in(".$ptstring.")";
}
//test result
if($flag==1){
if($testresult!="0"){
if($testresult=="HL"){
$testresult_condition=" and (result.AbnormalFlag='H' or result.AbnormalFlag ='L')";
$download_filename.="_abnormal";
}else{
$testresult_condition=" and result.AbnormalFlag='N'";
$download_filename.="_normal";
}
}
}
//age
if($age!="0"){
if($age=="0-30")
$age_condition=" and Visit.Age>=0 and Visit.Age<=30";
else if($age=="31-40")
$age_condition=" and Visit.Age>=31 and Visit.Age<=40";
else if($age=="41-50")
$age_condition=" and Visit.Age>=41 and Visit.Age<=50";
else if($age=="51-60")
$age_condition=" and Visit.Age>=51 and Visit.Age<=60";
else
$age_condition=" and Visit.Age>60";
}
$where=$year_condition.$centre_condition.$gender_condition.$test_condition.$testresult_condition.$age_condition;
return $where;
}
//echo "where condition ".preparewhere();
//echo "<br/>";
function preparequery(){
global $flag;
$selectquery="";
if($flag==1){
$selectquery="select Distinct(patient.PatName), Visit.VstMobile, Visit.VstDate, centres.SysField, Test.TestName, patient.Patsex, Visit.Age from Visit
inner join centres on Visit.RegistrationCentreCode=centres.SysNo
inner join patient on Visit.VstPatCode=patient.Patcode
inner join result on result.TrJobCode=Visit.VstCode
inner join Test on Test.TestCode=result.TrTestCode
inner join Param on Param.ParamCode=result.TrParamCode
";
}
else{
$selectquery="select Distinct(patient.PatName), Visit.VstMobile, Visit.VstDate, centres.SysField, profiles.ProfName, patient.Patsex, Visit.Age from Visit
inner join centres on Visit.RegistrationCentreCode=centres.SysNo
inner join patient on Visit.VstPatCode=patient.Patcode
inner join VisitProfile on VisitProfile.VstCode=Visit.VstCode
inner join profiles on profiles.ProfCode=VisitProfile.ProfCode
";
}
return $selectquery;
}
function strpos_offset($needle, $haystack, $occurrence) {
// explode the haystack
$arr = explode($needle, $haystack);
// check the needle is not out of bounds
switch( $occurrence ) {
case $occurrence == 0:
return false;
case $occurrence > max(array_keys($arr)):
return false;
default:
return strlen(implode($needle, array_slice($arr, 0, $occurrence)));
}
}
header("Content-type: text/csv");
header("Content-Disposition: attachment; filename=suburban_data.csv");
// Disable caching
header("Cache-Control: no-cache, no-store, must-revalidate"); // HTTP 1.1
header("Pragma: no-cache"); // HTTP 1.0
header("Expires: 0"); // Proxies
if(!mysqli_connect_errno())
{
$count=0;
$pcount=substr_count($_POST['profile_ids'], 'p'); //stores number of profiles from the string
$ptcount=substr_count($_POST['profile_ids'], ','); //stores number of tests or profiles selected
if($pcount==0)
$flag=1;
else if($pcount==$ptcount+1)
$flag=2;
else
$flag=3;
if($flag==1 || $flag ==2){ //if the selection contains either profiles or tests
//echo "profiles".$ptstring;
if($flag==1)
{
$tstring=$_POST['profile_ids'];
//echo "test".$tstring;
$mobile="";
$query_count_mobilenumbers=preparequery()." where ".preparewhere($tstring);
$con=mysqli_connect($db_host,$db_user,$db_password,$db_name);
$start = 0;
$limit = 1000;
do {
$mobile_result = mysqli_query(
$con,
$query_count_mobilenumbers." LIMIT {$start}, {$limit}",
MYSQLI_USE_RESULT // this always helps for this kind of processing (makes a smooth streaming)
);
$nbRows = 0; // cannot use mysqli_num_rows() because of `MYSQLI_USE_RESULT`
while($row = mysqli_fetch_array($mobile_result)) {
fputcsv($output, array($row[0],$row[1],$row[2],$row[3],$row[4],$row[5],$row[6]));
$nbRows ++;
}
// next batch
$start += $limit;
} while ($nbRows);
}
else{
$ptstring=str_replace("p","",$_POST['profile_ids']);
//echo "profiles are".$_POST['profile_ids'];
$query_count_mobilenumbers=preparequery()." where ".preparewhere($ptstring);
//echo "query is".$query_count_mobilenumbers;
$con=mysqli_connect($db_host,$db_user,$db_password,$db_name);
$start = 0;
$limit = 1000;
do {
$mobile_result = mysqli_query(
$con,
$query_count_mobilenumbers." LIMIT {$start}, {$limit}",
MYSQLI_USE_RESULT // this always helps for this kind of processing (makes a smooth streaming)
);
$nbRows = 0; // cannot use mysqli_num_rows() because of `MYSQLI_USE_RESULT`
while($row = mysqli_fetch_array($mobile_result)) {
fputcsv($output, array($row[0],$row[1],$row[2],$row[3],$row[4],$row[5],$row[6]));
$nbRows ++;
}
// next batch
$start += $limit;
} while ($nbRows);
}
}
else{ //if the selection contains both profiles and tests
//echo "inside both";
$no_of_profiles=substr_count($_POST['profile_ids'], 'p');
$nth_comma_index=strpos_offset(',',$_POST['profile_ids'], $no_of_profiles); //gets the nth comma's index
$pstring=str_replace("p","",substr($_POST['profile_ids'],0,$nth_comma_index));
$tstring=substr($_POST['profile_ids'],$nth_comma_index+1);
$mobile="";
$flag=1;
//query as tests
//echo "inside flag";
$query1=preparequery()." where ".preparewhere($tstring);
$con=mysqli_connect($db_host,$db_user,$db_password,$db_name);
//echo "quer".$query1;
$mobile_result1 = mysqli_query($con,$query1);
$start = 0;
$limit = 1000;
do {
$mobile_result1 = mysqli_query(
$con,
$query1." LIMIT {$start}, {$limit}",
MYSQLI_USE_RESULT // this always helps for this kind of processing (makes a smooth streaming)
);
$nbRows = 0; // cannot use mysqli_num_rows() because of `MYSQLI_USE_RESULT`
while($row = mysqli_fetch_array($mobile_result1)) {
fputcsv($output, array($row[0],$row[1],$row[2],$row[3],$row[4],$row[5],$row[6]));
$nbRows ++;
}
// next batch
$start += $limit;
} while ($nbRows);
$flag=2;
//echo "inside flag2";
$query2=preparequery()." where ".preparewhere($pstring);
$start = 0;
$limit = 1000;
do {
$mobile_result2 = mysqli_query(
$con,
$query2." LIMIT {$start}, {$limit}",
MYSQLI_USE_RESULT // this always helps for this kind of processing (makes a smooth streaming)
);
$nbRows = 0; // cannot use mysqli_num_rows() because of `MYSQLI_USE_RESULT`
while($row = mysqli_fetch_array($mobile_result2)) {
fputcsv($output, array($row[0],$row[1],$row[2],$row[3],$row[4],$row[5],$row[6]));
$nbRows ++;
}
$start += $limit;
} while ($nbRows);
}
}
else{
echo "failed to connect to database";
}
fclose($output);
?>
我敢打賭，您可以在服務器的錯誤日志中找到該錯誤，說明有關PHP腳本的內存不足。 發生這種情況的原因是，默認情況下，當從數據庫服務器接收到所有行時，對數據庫的查詢完成。 而且因為您要查詢數百萬行，所以結果集非常龐大，並且在PHP進程中占用大量內存。
我認為您應該使用MYSQLI_USE_RESULT作為對mysqli_query()調用的第三個參數：
$mobile_result = mysqli_query($con, $query_count_mobilenumbers, MYSQLI_USE_RESULT);
這使得mysqli_query()返回速度更快（可能是從服務器接收到結果的第一行之后），並且返回的結果集僅存儲所接收的行，直到程序使用mysqli_fetch_*()函數mysqli_fetch_*()進行檢索為止。
我從未使用過mysqli ，舊的mysql擴展為此目的具有單獨的功能（稱為mysql_unbuffered_query() ），但是mysql擴展已死，因此請勿使用它。
如果此方法仍然不起作用，則可以嘗試使用LIMIT從服務器批量獲取數據。
header("Content-type: text/csv");
header("Content-Disposition: attachment; filename=suburban_data.csv");
header("Cache-Control: no-cache, no-store, must-revalidate"); // HTTP 1.1
header("Pragma: no-cache"); // HTTP 1.0
header("Expires: 0"); // Proxies
// The next line is very important: it prevents PHP abruptly stop the script
// before it finishes what it's doing
set_time_limit(0);
$output = fopen("php://output", "w");
$headings=array("PatientName","Email","VisitDate","Centre","Profiles","Parameter","Age");
fputcsv($output, $headings);
$start = 0;
$limit = 1000;
do {
$mobile_result = mysqli_query(
$con,
$query_count_mobilenumbers." LIMIT {$start}, {$limit}",
MYSQLI_USE_RESULT // this always helps for this kind of processing (makes a smooth streaming)
);
$nbRows = 0; // cannot use mysqli_num_rows() because of `MYSQLI_USE_RESULT`
while($row = mysqli_fetch_array($mobile_result)) {
fputcsv($output, array($row[0],$row[1],$row[2],$row[3],$row[4],$row[5],$row[6]));
$nbRows ++;
}
// next batch
$start += $limit;
} while ($nbRows); // Stop when the query didn't return any row
fclose($output);
你可以試試ini_set('memory_limit', '-1');<\/strong> 在獲取\/選擇查詢之前的 PHP 腳本中，如果您使用共享主機，則在 php.ini 中設置post_max_size<\/strong> 。 ini_set('memory_limit', '-1');<\/strong> 對我來說工作得很好。
為了安全起見，您還可以設置內存大小而不是-1<\/strong> 。
從mysql數據庫中獲取數百萬條記錄，並將所有數據存儲到php中的單個數組中
[英]Fetching millions of record from mysql database and storing all data into single array in php
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.