如何从大量数据创建数组？

Question

I am selecting a lot of data from my mySQL database:

 $sql = "SELECT * FROM data WHERE id = ?";  
 $q = $pdo->prepare($sql);
 $q->execute([$id]);
 $array = $q->fetchAll(PDO::FETCH_ASSOC);

 var_dump($array);

I want to store this data into an array and after work with this array in a loop. My problem is now, that I have such an immense amount of data, that the array is loading and loading and my system is overwhelmed. Is there a way to create an array with a huge amount of data with a better performance?

Answer 1

The fetchAll() function maps the whole record to the variable.

Mapping one row per iteration will be considerably faster

$q->execute([$id]);

$i = 0;
while ($row = $q->fetch()) {
    // do something with $row
    $i++;
}

Answer 2

Please use fetch function instead of fetching all like below:

while ($arr = $stmt->fetch()){
   // do_other_stuff(); 
}

This will reduce the load on your system.

Please see this

Answer 3

I am currently processing 12,000 rows and encoding it into JSON arrays and in my opinion this is working best for me.

$sql = mysqli_query($dbconn, "SELECT * FROM data WHERE id = ?");

$rows = array();
    while($r = mysqli_fetch_assoc($sql)) {
    $rows[] = $r;
    }

Then echoing (or in my case echoing json_encode) it out.

Answer 4

The main thing to consider is:

1. Will this table data get increase further? (already big :))

Looping 500k records each time is not advisable/recommended. It definitely affects the speed/performance.

I suggest the following options:

1. Place a field in table to identify the highest folder (something like sort-order field) - update it correctly whenever needed. So that we can fetch the highest folder first and if there is a field to identify the parent then you can easily build a tree.
2. Think about a table to store highest folders with a foreign key to its main table. Update this accordingly so that you can easily identify highest folder.

PS: you can think of using a CRON to process the data and update the sort order if needed.