Select table from Sql server and insert data to Mysql table

Question

I have a running ms sql server and want some data copied to a mysql database.

i already can connect to them both so i made something like:

$pdo = new PDO('SQLSERVER', $user, $password);
$sql = "SELECT id, name FROM users";
$stmt = $pdo->prepare($sql);
$stmt->execute();

$json_users = array();
while ($row = $stmt->fetchObject()){
   $json_users[] = $row;
}

$pdo = new PDO('MYSQLDB', $user, $password);

foreach ($json_users as $key => $value){
    $sql = "INSERT INTO users (id, name) VALUES (:id, :name)"
    $stmt = $pdo->prepare($sql);
    $stmt->bindParam('id', $value->id, PDO::PARAM_INT);
    $stmt->bindParam('name', $value->name, PDO::PARAM_STR);
    $stmt->execute();
}

this does work but takes a lot of time cause its a big table.

so my question is can i insert the complete results from sqlserver to mysql at once with only one insert query? without the foreach?

Update: the table contains 173398 rows and 10 columns

Answer 1

With prepared statements (especially for multi-insert) you want to have your prepared statement outside your loop. You only need to set the query up once, then supply your data in each subsequent call

$sql = "INSERT INTO users (id, name) VALUES (:id, :name)";
$stmt = $pdo->prepare($sql);
foreach($json_users as $key => $value){
    $stmt->bindParam('id', $value->id, PDO::PARAM_INT);
    $stmt->bindParam('name', $value->name, PDO::PARAM_STR);
    $stmt->execute();
}

Answer 2

You can export this into CSV file first from MSSQL then import that file into MySQL.

$pdo = new PDO('SQLSERVER', $user, $password);
$sql = "SELECT id, name FROM users";
$stmt = $pdo->prepare($sql);
$stmt->execute();

$fp = fopen('/tmp/mssql.export.csv', 'w');
while ($row = $stmt->fetch(PDO::FETCH_NUM)){
   fputcsv($fp, array_values($row));
}
fclose($fp);
$pdo = new PDO('MYSQLDB', $user, $password, array(PDO::MYSQL_ATTR_LOCAL_INFILE => 1));

$sql = <<<eof
    LOAD DATA LOCAL INFILE '/tmp/mssql.export.csv'
     INTO TABLE user_copy
     FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
     LINES TERMINATED BY '\n'
    (id,name)
eof;
$pdo->exec($sql);

one drawback with the above, you need to have this configuration enabled in my.cnf ( MySQL configuration )

    [server]
    local-infile=1

Since MySQL cannot read files that are owned by others unless it it opened with --local-infile=1

Answer 3

I would suggest not bind values, but generate the query string:

$sql = "";
foreach ($json_users as $key => $value){
    if ($sql=="") {
       $sql = "INSERT INTO users (id, name) VALUES ";
       $sql =." (".$value->id.',"'.$value->name.'")';
    } else {
       $sql .= ", (".$value->id.',"'.$value->name.'")';
    }
}

$stmt = $pdo->prepare($sql);
$stmt->execute();

It is not best practice, but since you trust data source it could help.

Answer 4

Consider doing bulk inserts instead of one row at a time.

$sourcedb = new \PDO('SQLSERVER', $sourceUser, $sourcePassword);
$targetdb = new \PDO('MYSQLDB', $targetUser, $targetPassword);

$sourceCountSql = "SELECT count(*) count FROM users;";

/**
 * for mssql server 2005+ 
 */
$sourceSelectSql = "
SELECT
    id,
    name
FROM
    (
        SELECT
            ROW_NUMBER() OVER (ORDER BY id) RowNum,
            id,
            name
        FROM
            users
    ) users
WHERE
    RowNum >= %d
    AND RowNum < %d
ORDER BY
    RowNum
";

/**
 * for mssql server 2012+ 
 */
$sourceSelectSql = "
SELECT 
    FROM TableName ORDER BY id OFFSET 10 ROWS FETCH NEXT 10 ROWS ONLY;
SELECT
    id,
    name
FROM
    users
ORDER BY
    id
OFFSET %d ROWS
FETCH NEXT %d ROWS ONLY        
";    

$sourceCount = $sourcedb->query($sourceCountSql, \PDO::FETCH_COLUMN, 0);

$rowCount = 1000;

$count = 0;
$count2 = 0;

for($x = 0; $x < $sourceCount; $x += $rowCount) {

    $sourceRecords = $sourcedb->query(sprintf($sourceSelectSql, $x, $rowCount), \PDO::FETCH_ASSOC);

    $inserts = [];

    foreach($sourceRecords as $row) {
        $inserts[] = sprintf("(:id_%1$d, :name_%1$d)", $count++);
    }

    $stmt = $targetdb->prepare(sprintf("INSERT INTO users (id, name) VALUES %s;", implode(',', $inserts));

    foreach($sourceRecords as $row) {
        $stmt->bindParam(sprintf('id_%d', $count2), $row['id'], \PDO::PARAM_INT);
        $stmt->bindParam(sprintf('name_%d', $count2), $row['name'], \PDO::PARAM_STR);
        $count2++;
    }

    $targetdb->execute();

    unset($inserts);

}