Slow import of data in mysql using mysqlimport in version 5.7.31 on Ubuntu 18.04 Im loading genetic data from Ensembl and following the directions here https://m.ensembl.org/info/docs/webcode/mirror/install/ensembl-data.html .
this is the command I am using
/data/mysql/bin/mysqlimport -u mysqldba --fields-terminated-by='\t' --fields-escaped-by=\\ homo_sapiens_core_100_38 -L allele_synonyms.txt
A few of the tables are massive. 40GB, 90GB and ~300GB. I didn't expect this to go fast but it seems to be going way too slow. I can import 10GB tables in an hour but this 40GB one is killing me. I even used slice to make small chunks and load them but at 20GB it seems to get infinitely longer. Like days to load the data and that's when I add just one more 5GB chunk after 20GB is loaded.
I have followed other suggestions like this post here Improve performance of mysql LOAD DATA / mysqlimport?
I have turned off slow query logging in mysql.cnf and tried other suggestions found on Stack Overflow like increasing the buffers.
Im using a very large space for my temp files I'm specifically using mysqlimport on a txt.file The 40GB file has like 800,000,000 Rows so its a lot of data. I have tried adding --num-threds=4 The machine has decent specs 4 cores and 14GB ram
Im not sure if this is the correct answer but this is what worked for me. My data was like 99.9% read so I was trying to us MyISAM for the massive tables. By simply changing the tables to InnoDB it worked. I did this for any table over 25GB and it worked.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.