get_file_content（）耗尽了允许的内存大小（读取.dat文件）

Question

I am in trouble while running my php script.

I am try to read data from a .dat file by using get_file_content()

Here is my code:

$file = 'abc.dat'; $data = file_get_contents($file);

But while it giving me error like

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 2097152 bytes) in /....

I tried many solutions for it like ini_set('memory_limit', '128M'); , init_set('maximum_exec_time', '0');

But IT's not working for me....

I have uploaded my php file on one of webhost. While Everything works smooth when i run the script on localhost.

Can anyone help me please...?? Is there any other way for read my long file.???

Edited:

Yes, i need realy need whole data in one array.

The concept behind code is Face Detection and comparison in PHP. Here is the code for Face Detection, PLEASE GIVE ME ANY SUGGESTiON..!!

<?php


ini_set('max_execution_time', 123456);

class FaceDetector {

protected $detection_data;
protected $canvas;
protected $face;
private $reduced_canvas;

public function __construct($detection_file = 'detection1.dat')
{
    if (is_file($detection_file)) {

        $this->detection_data = file_get_contents($detection_file);

    } else {
        throw new Exception("Couldn't load detection data");
    }
}


public function faceDetect($file)
{
    if (is_resource($file)) {

        $this->canvas = $file;

    } elseif (is_file($file)) {

        $this->canvas = imagecreatefromjpeg($file);

    } else {

        throw new Exception("Can not load $file");
    }

    $im_width = imagesx($this->canvas);
    $im_height = imagesy($this->canvas);

    //Resample before detection?
    $ratio = 0;
    $diff_width = 320 - $im_width;
    $diff_height = 240 - $im_height;
    if ($diff_width > $diff_height) {
        $ratio = $im_width / 320;
    } else {
        $ratio = $im_height / 240;
    }

    if ($ratio != 0) {
        $this->reduced_canvas = imagecreatetruecolor($im_width / $ratio, $im_height / $ratio);

        imagecopyresampled(
            $this->reduced_canvas,
            $this->canvas,
            0,
            0,
            0,
            0,
            $im_width / $ratio,
            $im_height / $ratio,
            $im_width,
            $im_height
        );

        $stats = $this->getImgStats($this->reduced_canvas);

        $this->face = $this->doDetectGreedyBigToSmall(
            $stats['ii'],
            $stats['ii2'],
            $stats['width'],
            $stats['height']
        );

        if ($this->face['w'] > 0) {



            $this->face['x'] *= $ratio;
            $this->face['y'] *= $ratio;
            $this->face['w'] *= $ratio;

        }
    } else {
        $stats = $this->getImgStats($this->canvas);


        $this->face = $this->doDetectGreedyBigToSmall(
            $stats['ii'],
            $stats['ii2'],
            $stats['width'],
            $stats['height']
        );
    }

    return ($this->face['w'] > 0);
}


public function toJpeg()
{
    $color = imagecolorallocate($this->canvas, 255, 0, 0); //red

    imagerectangle(
        $this->canvas,
        $this->face['x'],
        $this->face['y'],
        $this->face['x']+$this->face['w'],
        $this->face['y']+ $this->face['w'],
        $color
    );
    $image = imagecreatefromjpeg('lena512color.jpg');
    $filename = './3/cropped_whatever1.JPG';
    //$original=imagejpeg($this->canvas);
    $cropped = imagecreatetruecolor($this->face['w']+5, $this->face['w']+5);
    //imagecopy($cropped, $original, 0, 0, $this->face['x'],$this->face['y'], $this->face['x']+$this->face['w'], $this->face['y']+ $this->face['w']);
    header('Content-type: image/jpeg');

    $im_width = imagesx($this->canvas);
    $im_height = imagesy($this->canvas);

    $start_x=$im_width-($im_width-$this->face['x']);
    $start_y=$im_height-($im_height-$this->face['y']);

    $result=imagecopy ( $cropped , $image , 0 , 0, $start_x , $start_y , $this->face['w']+5, $this->face['w']+5);
    //print $result;
    imagejpeg($cropped);
    imagejpeg($cropped, $filename, 100);
}

public function toJson()
{
    return json_encode($this->face);
}

public function getFace()
{
    return $this->face;
}

protected function getImgStats($canvas)
{
    $image_width = imagesx($canvas);
    $image_height = imagesy($canvas);
    $iis =  $this->computeII($canvas, $image_width, $image_height);
    return array(
        'width' => $image_width,
        'height' => $image_height,
        'ii' => $iis['ii'],
        'ii2' => $iis['ii2']
    );
}

protected function computeII($canvas, $image_width, $image_height)
{
    $ii_w = $image_width+1;
    $ii_h = $image_height+1;
    $ii = array();
    $ii2 = array();

    for ($i=0; $i<$ii_w; $i++) {
        $ii[$i] = 0;
        $ii2[$i] = 0;
    }

    for ($i=1; $i<$ii_h-1; $i++) {
        $ii[$i*$ii_w] = 0;
        $ii2[$i*$ii_w] = 0;
        $rowsum = 0;
        $rowsum2 = 0;
        for ($j=1; $j<$ii_w-1; $j++) {
            $rgb = ImageColorAt($canvas, $j, $i);
            $red = ($rgb >> 16) & 0xFF;
            $green = ($rgb >> 8) & 0xFF;
            $blue = $rgb & 0xFF;
            $grey = (0.2989*$red + 0.587*$green + 0.114*$blue)>>0;  // this is what matlab uses
            $rowsum += $grey;
            $rowsum2 += $grey*$grey;

            $ii_above = ($i-1)*$ii_w + $j;
            $ii_this = $i*$ii_w + $j;

            $ii[$ii_this] = $ii[$ii_above] + $rowsum;
            $ii2[$ii_this] = $ii2[$ii_above] + $rowsum2;
        }
    }
    return array('ii'=>$ii, 'ii2' => $ii2);
}

protected function doDetectGreedyBigToSmall($ii, $ii2, $width, $height)
{
    $s_w = $width/20.0;
    $s_h = $height/20.0;
    $start_scale = $s_h < $s_w ? $s_h : $s_w;
    $scale_update = 1 / 1.2;
    for ($scale = $start_scale; $scale > 1; $scale *= $scale_update) {
        $w = (20*$scale) >> 0;
        $endx = $width - $w - 1;
        $endy = $height - $w - 1;
        $step = max($scale, 2) >> 0;
        $inv_area = 1 / ($w*$w);
        for ($y = 0; $y < $endy; $y += $step) {
            for ($x = 0; $x < $endx; $x += $step) {
                $passed = $this->detectOnSubImage($x, $y, $scale, $ii, $ii2, $w, $width+1, $inv_area);
                if ($passed) {
                    return array('x'=>$x, 'y'=>$y, 'w'=>$w);
                }
            } // end x
        } // end y
    }  // end scale
    return null;
}

protected function detectOnSubImage($x, $y, $scale, $ii, $ii2, $w, $iiw, $inv_area)
{
    $mean  = ($ii[($y+$w)*$iiw + $x + $w] + $ii[$y*$iiw+$x] - $ii[($y+$w)*$iiw+$x] - $ii[$y*$iiw+$x+$w])*$inv_area;

    $vnorm = ($ii2[($y+$w)*$iiw + $x + $w]
              + $ii2[$y*$iiw+$x]
              - $ii2[($y+$w)*$iiw+$x]
              - $ii2[$y*$iiw+$x+$w])*$inv_area - ($mean*$mean);

    $vnorm = $vnorm > 1 ? sqrt($vnorm) : 1;

    $passed = true;
    for ($i_stage = 0; $i_stage < count($this->detection_data); $i_stage++) {
        $stage = $this->detection_data[$i_stage];
        $trees = $stage[0];

        $stage_thresh = $stage[1];
        $stage_sum = 0;

        for ($i_tree = 0; $i_tree < count($trees); $i_tree++) {
            $tree = $trees[$i_tree];
            $current_node = $tree[0];
            $tree_sum = 0;
            while ($current_node != null) {
                $vals = $current_node[0];
                $node_thresh = $vals[0];
                $leftval = $vals[1];
                $rightval = $vals[2];
                $leftidx = $vals[3];
                $rightidx = $vals[4];
                $rects = $current_node[1];

                $rect_sum = 0;
                for ($i_rect = 0; $i_rect < count($rects); $i_rect++) {
                    $s = $scale;
                    $rect = $rects[$i_rect];
                    $rx = ($rect[0]*$s+$x)>>0;
                    $ry = ($rect[1]*$s+$y)>>0;
                    $rw = ($rect[2]*$s)>>0;
                    $rh = ($rect[3]*$s)>>0;
                    $wt = $rect[4];

                    $r_sum = ($ii[($ry+$rh)*$iiw + $rx + $rw]
                              + $ii[$ry*$iiw+$rx]
                              - $ii[($ry+$rh)*$iiw+$rx]
                              - $ii[$ry*$iiw+$rx+$rw])*$wt;

                    $rect_sum += $r_sum;
                }

                $rect_sum *= $inv_area;

                $current_node = null;

                if ($rect_sum >= $node_thresh*$vnorm) {

                    if ($rightidx == -1) {

                        $tree_sum = $rightval;

                    } else {

                        $current_node = $tree[$rightidx];

                    }

                } else {

                    if ($leftidx == -1) {

                        $tree_sum = $leftval;

                    } else {

                        $current_node = $tree[$leftidx];
                    }
                }
            }

            $stage_sum += $tree_sum;
        }
        if ($stage_sum < $stage_thresh) {
            return false;
        }
    }
    return true;
}

}

Answer 1

You you really need all of the file loaded into memory at the same time? If so, then you need to talk to your host about getting your memory limit increased, and they will probably tell you to upgrade your plan or move to a VPS.

Ideally, you should be working with you data in the smallest chunks for whatever you're doing. eg:

$myfile = '/path/to/file.dat';
$processed_data = array();

if( ! $fh = fopen($myfile, 'r') ) { die("could not open $myfile."); }

while( $line = fgets($fh) ) {
  $processed_data[] = processing_function($line);
}
fclose($fh);

Where processing_function() is only returning the exact data you need, thus keeping your memory usage to a minimum.

If you posted the code that details what you're actually doing with this data people might be able to suggest a better way to go about it.

Answer 2

It seems like your memory_limit is set to 64MB. Can you confirm that your memory limit is indeed 128M when the script is running:

var_dump(ini_get('memory_limit'));

If it is not 128M, what hosting are you using? Usually this can be upped by htaccess:

php_value memory_limit 128M

If that doesn't fix it, you will need to contact your hosting provide (we can't help you with exact ode if you don't want to disclose the name of your hosting provider) to up the memory limit.

Answer 3

You can use buffer like here: Alternative way to read raw I/O stream in PHP If you could process the data in the while cycle, then:

$f = fopen('abc.dat', 'r');
$data = '';
while ($buffer =  fread($f, 8192)) do_some_processing($buffer);
fclose($f);