PHP Get X Amount of Lines from File until Done

Question

I have a text file with 2000 lines and need to get 100 at a time in a continuous loop. I can get it to work, but I have to manually change $i value for each chunk of 100. Here is the code:

    $file = 'postcode_2000.txt';
    for ($i = 0; $i <= 99; $i++) {

        $str = str_replace(PHP_EOL, '', $file[$i]);
        $list[] = $str;

    }

    $list_json = json_encode($list);

How can I iterate over this getting the next 100 each time?

Answer 1

Once you have the file in an array you can use array_chunk() . First read the file into an array:

$lines = 'postcode_2000.txt';

Then chunk the file into 100 line parts:

$chunks = array_chunk($lines, 100, true); // true keeps a set of consecutive line numbers

The output will be something like this:

Array
(
    [0] => Array
        (
            [0] => line in file
            [1] => line in file
            [2] => line in file
            [3] => line in file
            [4] => line in file
            ...
            [94] => line in file
            [95] => line in file
            [96] => line in file
            [97] => line in file
            [98] => line in file
            [99] => line in file
        )

    [1] => Array
        (
            [100] => line in file
            [101] => line in file
            [102] => line in file
            [103] => line in file
            [104] => line in file
            ...
            [194] => line in file
            [195] => line in file
            [196] => line in file
            [197] => line in file
            [198] => line in file
            [199] => line in file
        )

    [2] => Array
        (
            [200] => line in file
            [201] => line in file
            [202] => line in file
            [203] => line in file
            [204] => line in file
            ...
            [294] => line in file
            [295] => line in file
            [296] => line in file
            [297] => line in file
            [298] => line in file
            [299] => line in file
        )
    ...    
    [18] => Array
        (
            [1800] => line in file
            [1801] => line in file
            [1802] => line in file
            [1803] => line in file
            [1804] => line in file
            ...
            [1894] => line in file
            [1895] => line in file
            [1896] => line in file
            [1897] => line in file
            [1898] => line in file
            [1899] => line in file
        )

    [19] => Array
        (
            [1900] => line in file
            [1901] => line in file
            [1902] => line in file
            [1903] => line in file
            [1904] => line in file
            ...
            [1994] => line in file
            [1995] => line in file
            [1996] => line in file
            [1997] => line in file
            [1998] => line in file
            [1999] => line in file
        )

)

Answer 2

If you really want to use loops instead of array_chunk :)

<?php

$file = file('/tmp/postcodes_2000.txt');

$list = array();
$line_count = count($file);
$chunk_size = 100;
$current_line = 0;
echo "Line count: ${line_count}" . PHP_EOL;
while( $current_line < $line_count ) {
  $i = 0;
  $chunk = array();
  while($i < $chunk_size) {
    $str = str_replace(PHP_EOL, '', $file[$current_line]);
    $chunk[] = $str;
    echo "> Line ${current_line} : " . $file[$current_line] . PHP_EOL;
    $i++;
    $current_line++;
  }
  $list[] = $chunk;
}

$list_json = json_encode($list);
echo $list_json . PHP_EOL;

Answer 3

Generators are great for when you are calculating large sets and you don't want to allocate memory for all of the results at the same time or when you don't know if you will need all of the results, Due to the way results are processed, the memory footprint can be reduced to a very bare minimum by allocating memory for only the current result.

So this is a very fast way which consume less memory to achieve your goal:

we open the file ,read a precise amount of lines and then yield the result while keeping the state for the next iteration and thus allow you to handle your chunks very fast and with less memory consumption.

Proceeding this way has the advantage to avoid reading all the file twice to build chunks (read it one time to build a big array containing all the lines and then loop through this array to build chunks) before allowing you finally (a third time) to loop through the array of chunks...

function file_get_x_lines($file,$amount=100){
    if(!file_exists($file)||!is_file($file)||!is_readable($file))
        return;
    if(!is_int($amount)||$amount<1) $amount=1;
    $handle=fopen($file,'rb');
        $i=0;
        while($line=fgets($handle)){
            if($i===0)
                $tmp=array();
            $chunks[]=rtrim($line,PHP_EOL);
            $i++;
            if($i===$amount){
                $i=0;
                $tmp=$chunks;
                $chunks=array();
                yield $tmp;
            }
        }
        if(!$line)
            yield $chunks; 

    }

then you can use it like you would use any generator

foreach(file_get_x_lines(__FILE__) as $list)    {
    $list=json_encode($list);
    //do stuff
}

if you want to keep same keys as the line number you can lightly alter the function this way:

function file_get_x_lines($file,$amount=100){
        if(!file_exists($file)||!is_file($file)||!is_readable($file))
            return;
        if(!is_int($amount)||$amount<1) $amount=1;
        $handle=fopen($file,'rb');
            $i=0;
            $j=1;
            while($line=fgets($handle)){
                if($i===0)
                     $tmp=array();
                $chunks[$j]=rtrim($line,PHP_EOL);
                $i++;
                $j++;
                if($i===$amount){
                    $i=0;
                    $tmp=$chunks;
                    $chunks=array();
                    yield $tmp;
                }
            }
            if(!$line)
                yield $chunks; 

}