用PHP处理大文件

Question

What is the best way of processing very large files in php. This is my current the scenario:

1. I extract a raw file from a Network Mgmt System (NMS) about all parameters of all network elements (NMS is running in a UNIX box).
2. I FTP the raw file in my PC box using PHP.
3. I process the raw file line by line using the PHP's fget() function.
4. Every line i use string matching and regexp matching to extract necessary data until i am able to compose 1 line of necessary data separated by commas (",").
5. I repeat step 4 until EOF and have the full CSV file.
6. I then throw this data to my database using sql's "LOAD DATA INFILE"

My problem now is, I have 1 raw file that reaches more/less 200MB, and has more/less 180 columns of data and because of these my php script cannot finish proccessing the whole file because upon processing it is exhausting all the 1024MB memory I allocate on my php.ini file.

Hoping to have recomendations on the best workaround of this problem. Thanks!

code of the processing part below:

while( !feof($fh) ){
set_time_limit(0);
$l_buffer = fgets( $fh, $fsize );
$l_LineStream = explode( ' ', trim( $l_buffer ) );
$l_FilteredLineStream = array_filter( $l_LineStream, array( $this, 'RemoveEmptyElement' ) );
$l_GrepMatchArray = preg_grep( '/^BSC.*_.*$/', $l_FilteredLineStream );
if( count( $l_GrepMatchArray ) > 0 ){
    foreach( $l_GrepMatchArray as $l_BSCFound ){
        $l_BSCFound = explode( '_', $l_BSCFound );
        $l_BSCHoming = $l_BSCFound[1];
    }
}
$l_GrepMatchArray = preg_grep( '/^BTS-[0-9]*$/', $l_FilteredLineStream );
if( count( $l_GrepMatchArray ) > 0 ){
    foreach( $l_GrepMatchArray as $l_BTSFound ){
        $l_CurrBTS = $l_BTSFound;
    }
}
/**/
if( $l_PrevBTS != $l_CurrBTS && isset( $l_BTSArray ) && count( $l_BTSArray ) > 0 ){
    #$this->BTS_Array[] = $l_BTSArray;
    if( $l_FirstLoop == true ){
        $this->WriteDataToCSVFile( $l_BTSArray, $l_FilePath, true );
        $l_FirstLoop = false;
    }else{
        $this->WriteDataToCSVFile( $l_BTSArray, $l_FilePath );
    }
}
/**/
if( count( $l_GrepMatchArray ) > 0 ){
    #var_dump( $l_FilteredLineStream );
    $l_BTSArray = $this->InstantiateEmptyBTSArray();
    #$l_BTSArray['CI'] = '';
    $l_BTSArray['BSC'] = $l_BSCHoming;
    $l_BTSArray['BCF'] = $l_FilteredLineStream[0];
    $l_BTSArray['BTS'] = $l_FilteredLineStream[3];
    $l_BTSArray['CELL NAME'] = $l_FilteredLineStream[6];
}
if( $l_GetPLMNNextLineData == true && isset( $l_BTSArray['PLMN'] ) ){
    $l_BTSArray['PLMN'] .= trim( $l_buffer );
    $l_GetPLMNNextLineData = false;
}
$l_GrepMatchArray = preg_match( '/\.\(.*$/', $l_buffer, $reg_match );

if( count( $reg_match ) > 0 ){
    $l_KeyName = substr( $reg_match[0], 2, strpos( $reg_match[0], ')' ) - 2 );
    preg_match( '/[[:space:]].*|[-].*/', $reg_match[0], $param_value );
    $l_BTSArray[$l_KeyName] = trim( $param_value[0] );
    if( $l_KeyName == 'PLMN' ){
        $l_GetPLMNNextLineData = true;
    }
}
$l_PrevBTS = $l_CurrBTS;
}

Answer 1

You should check if your script is really processing the big file line by line (one line at a time).

• do you keep a read line in an array?
• do you write the CSV line instantly in your file, or do you keep all the generated lines in an array?
• etc.

If you process the file line by line, it should not use 1GB+ memory.

Answer 2

Why do you save to MySQL only in the end of process? When you parsed a line flush that to the database, so you will just use few MB for each rows.

To address the comment:

You can use INSERT DELAYED to let database manage the load, and to don't stress it too much

Answer 3

If you are running out of 1024MB of memory processing a 200MB file then I would suggest you have a memory issue somewhere. I would suggest you check your code for areas that could be holding on to resources that are no longer required.