如何使用 jdbc 更快地将大量数据从 tsv 导入到 MySQL？

Question

I have two huge tsv files(10mil records) where tsv one file has the attributes id, name, age and other had attributes id, email and phno. I tried to read the first file and insert the records into the Person table and then read the second file and update the Person table. This approach takes time as the table is first inserted with 10 mil records and then they are updated. IS there any other way to speed this process? PS some Id are not there in the 2nd tsv file so I was not able to merge both of them.

Answer 1

Why you don't try LOAD DATA INFILE which is a highly optimized, MySQL -specific statement that directly inserts data into a table from a CSV / TSV file.

There are two ways to use LOAD DATA INFILE . You can copy the data file to the server's data directory (typically /var/lib/mysql-files/ ) and run:

LOAD DATA INFILE '/path/to/products.csv' INTO TABLE products;

Or you can also store the data file on the client side, and use the LOCAL keyword:

LOAD DATA INFILE '/path/to/products.csv' INTO TABLE products;

High-speed inserts with MySQL

You should also check MySql Documentation - LOAD DATA Statement And you could use a statement like this one:

LOAD DATA INFILE 'data.txt' 
  INTO TABLE tbl_name
  FIELDS TERMINATED BY ',' ENCLOSED BY '"'
  LINES TERMINATED BY '\r\n'
  IGNORE 1 LINES;