在 Ubuntu 18.04 上使用 mysqlimport 在版本 5.7.31 中导入 mysql 中的数据缓慢

Question

Slow import of data in mysql using mysqlimport in version 5.7.31 on Ubuntu 18.04 Im loading genetic data from Ensembl and following the directions here https://m.ensembl.org/info/docs/webcode/mirror/install/ensembl-data.html .

this is the command I am using

/data/mysql/bin/mysqlimport -u mysqldba --fields-terminated-by='\t' --fields-escaped-by=\\ homo_sapiens_core_100_38 -L allele_synonyms.txt

A few of the tables are massive. 40GB, 90GB and ~300GB. I didn't expect this to go fast but it seems to be going way too slow. I can import 10GB tables in an hour but this 40GB one is killing me. I even used slice to make small chunks and load them but at 20GB it seems to get infinitely longer. Like days to load the data and that's when I add just one more 5GB chunk after 20GB is loaded.

I have followed other suggestions like this post here Improve performance of mysql LOAD DATA / mysqlimport?

I have turned off slow query logging in mysql.cnf and tried other suggestions found on Stack Overflow like increasing the buffers.

Im using a very large space for my temp files I'm specifically using mysqlimport on a txt.file The 40GB file has like 800,000,000 Rows so its a lot of data. I have tried adding --num-threds=4 The machine has decent specs 4 cores and 14GB ram

Answer 1

Im not sure if this is the correct answer but this is what worked for me. My data was like 99.9% read so I was trying to us MyISAM for the massive tables. By simply changing the tables to InnoDB it worked. I did this for any table over 25GB and it worked.