How to insert million rows into MySQL database using php script?

Question

I`m looking for php script to generate random data and insert 1 million rows into the mysql database.

It`sa test case i received as part of my homework.

Here is a piece of code i`ve tried but it didnt work as the allowed memory size exhausted.

PS Cant change memory_limit as it`s prohibited and counts as cheating :)

$query = "INSERT INTO book(firstname, lastname, phone_number, address) VALUES ";
   $query_parts = array();
   for($i=0; $i<1000000; $i++){
     $firstname = 'name' . $i;
     $lastname = 'lastname' . $i;
     $phone_number = mt_rand();
     $address = 'address' . mt_rand();
     $query_parts[] = "('" . $firstname . "', '" . $lastname . "', '" . $phone_number . "', '" . $address . "')";
   }
   $query .= implode(',', $query_parts);

   $create_record_query = mysqli_query($connection, $query);

Answer 1

Do it in batches, eg 1000 rows at a time.

Use PDO rather than mysqli, as it allows you to execute a prepared statement with an array to supply the values.

$batch_size = 1000;

$sql = "INSERT INTO book(firstname, lastname, phone_number, address) VALUES ";
// Add 1,000 value lists
$sql .= str_repeat("(?,?,?,?),", $batch_size-1);
$sql .= "(?,?,?,?)";
$stmt = $pdo->prepare($sql);
for ($i = 0; $i < (1000000/$batch_size); $i++) {
    $vals = array();
    for ($j = 0; $j < $batch_size; $j++) {
        array_push($vals, "name$j", "address$j", mt_rand(), "address" . mt_rand());
    }
    $stmt->execute($vals);
}

You might need to increase the MySQL server's max_allowed_packet setting to handle 1000 rows, or decrease the batch size. Make sure that $batch_size evenly divides into 1 million -- the code above doesn't handle a last batch that's not the same size (consider fixing that as an exercise for the reader).