Memory problems while copying data from one mysql table to another

Question

I need to fetch, convert and then insert 4 000 000 rows into another table. Needles to say, a memory exhausted error kicks in after 300 000~ entries. After every loop allocated memory increases by exactly 160 bytes. I know that using mysql_unbuffered_query() would be possible, although it forces me to fetch all result rows before I can execute another query, which ends up in memory exhausted error again. So, what's the best way to do this in a single run?

mysql_connect($host, $user, $password);
mysql_select_db($db);

$getOldData = mysql_query("
SELECT *
FROM players_online 
ORDER by id ASC
");
$numRows = mysql_num_rows($getOldData);

for ($i=0; $i < $numRows; $i++) { 
$oldData = mysql_fetch_assoc($getOldData);
$hour = explode(':', $oldData['hour']);
$quarters = $hour[0] * 4 + $hour[1] / 15;

$update = mysql_query("
    INSERT INTO players_online_2 (world_id, players_online, quarters_past_midnight, date)
    VALUES (
        '".$oldData['world_id']."', 
        '".$oldData['players_online']."', 
        '".$quarters."', 
        '".$oldData['date']."'
    )
    ON DUPLICATE KEY UPDATE
        world_id='".$oldData['world_id']."', 
        players_online='".$oldData['players_online']."', 
        quarters_past_midnight='".$quarters."', 
        date='".$oldData['date']."'
");

if (mysql_error()) {
    echo mysql_error();
    die();
}

echo memory_get_usage().PHP_EOL;

}

Answer 1

MySQL Workbench will let you export the old database and import it into the new location.

With that said though, if you want to do this in PHP, you should probably not return the entire table in one shot. You could use LIMIT and OFFSET to split the query into pages, and then execute one page at a time. Something like this:

for ($i = 0; $i < $Limit; $i++)
{
  // SELECT * FROM players_online ORDER by id ASC LIMIT 1000 OFFSET ' . $i
}

Answer 2

You can do this in a single query, something like:

INSERT INTO players_online_2 (world_id, players_online, quarters_past_midnight, date)
    select world_id, players_online, quarters_past_midnight, date
    FROM players_online po
    ORDER by id ASC
ON DUPLICATE KEY UPDATE
    world_id = po.world_id,
    players_online = po.players_online, 
    quarters_past_midnight = po.quarters_past_midnight, 
    date = po.date;