MySQL从一个表到另一个PHP传输8000+行

Question

I am working on a script in which I want to transfer 8000+ rows from one table to another table in another database with selected columns. By far I have thought to fetch the selected columns of the entire source table using PDOStatement::fetchAll() as a $result array and then prepare a mySQL insert statement using PDO::prepare() method as

INSERT INTO targetTable (targetColumn1,targetColumn2...) VALUES 
(sourceRow1ForColumn1,sourceRow1ForColumn2,..),
(sourceRow2ForColumn1,sourceRow2ForColumn2,..),
(sourceRow3ForColumn1,sourceRow3ForColumn2,...),...

But I think that this is not an efficient way and there should be an optimal way to do what I want to do. Is there any better way I can do what I want to do?

Answer 1

If it's going to be done one time, you do not need to care of the performance.

put set_time_limit(0); at the beginning of your PHP file.

Then, connect to your first DB. read the data from the source table and store in an array.

Now, disconnect and connect to the new DB and insert the array data to the target table

And for caring the duplicate keys, you can use INSERT OR UPDATE ON DUPLICATE in your query.

If your records are going to be huge, you might have problem with the memory. So, it's better to get the count before storing into the array. Then, do the process in a loop and at the end of each loop, free up the array by unset(). So, you split the records and keep your memory usage in a proper way.

Answer 2

The optimal way to do this would be to use an INSERT ... SELECT statement.

INSERT INTO targetTable (targetColumn1,targetColumn2)
SELECT expr1, expr2 FROM one_table

expr1 and expr2 represent expressions that return the values that you want to insert into targetColumn1 and targetColumn2 columns, respectively.

The expressions could be as simple reference to a colum name in the source table.

---

If the tables are in different databases (schemas) on the same MySQL instance, you can qualify the table name with the name of the database. (An unqualified table name refers to a table in the current database (ie USE mydatabase.)

If your current database is the target database, qualify the name table in the FROM clause

INSERT INTO targetTable (targetColumn1,targetColumn2)
SELECT expr1, expr2 FROM anotherdb.one_table
                         ^^^^^^^^^^

If the current database contains the source table, qualify the name of the targetTable.

Or just qualify both table names, and it doesn't matter what database is current.

---

Just connect to MySQL as a user that has sufficient privileges on each table.

GRANT SELECT ON anotherdb.one_table TO myuser@'%';
GRANT INSERT ON targetdb.targetTable TO myuser@'%';

---

If the insert attempts to insert a row that violates a unique constraint in the targetTable, the same thing would happen with an INSERT ... SELECT as would with the INSERT ... VALUES statement... the statement will fail.

If you want to disregard errors about duplicates, and just skip over those rows, then use an INSERT IGNORE ... SELECT atatement. (The IGNORE keywords causes MySQL not to skip over the row, rather than throwing an error when a unique key constraint is violated.

Or, you could write the SELECT statement to exclude rows that already exist in the target table.