Process large csv file using mysql load data infile

Question

I am trying to read CSV file with 500k rows and 81 columns and insert it in database. CSV size is 160 MB. I have to process it every 3-4 hours. Every 3-4 hours I will have a new CSV with some new records and some existing. For that I have followed many methods but nothing working perfectly.

Method 1 : It reads all records but takes too much time. It took more than 1 hour to read 200k records and then time out.

$file    = fopen($path, 'r');
while (($line = fgetcsv($file)) !== FALSE) {
}

As a solution: I can increase php limit, excution time and memory limit to avoid this error but I feel it is very slow.

Method 2:

$query = "LOAD DATA LOCAL INFILE products.csv INTO TABLE tmp_tbl LINES TERMINATED BY '\\r\\n' FIELDS TERMINATED BY ',' IGNORE 1 LINES";

DB::select($query);

It gave following error.

SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'FIELDS TERMINATED BY ',' IGNORE 1 LINES' at line 1 (SQL: LOAD DATA LOCAL INFILE 'products.csv' INTO TABLE tmp_tbl LINES TERMINATED BY '\\r\\n' FIELDS TERMINATED BY ',' IGNORE 1 LINES)

Method 3:

$query = "LOAD DATA LOCAL INFILE 'products.csv' INTO TABLE tmp_tbl LINES TERMINATED BY '\\r\\n' FIELDS TERMINATED BY ',' IGNORE 1 LINES";

DB::connection()->getpdo()->exec($query);

or

$query = "LOAD DATA LOCAL INFILE '".$file_path."' INTO TABLE tmp_tbl LINES TERMINATED BY '\\n' FIELDS TERMINATED BY ',' IGNORE 1 LINES";

DB::connection()->getpdo()->exec($query);

Got following error

SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'FIELDS TERMINATED BY ',' IGNORE 1 LINES' at line 1

Method 4

$query = "LOAD DATA LOCAL INFILE '".$file_path."' INTO TABLE tmp_tbl";

DB::connection()->getpdo()->exec($query);

Got following error

PDO::exec(): MySQL server has gone away

Method 5

$query = "LOAD DATA LOCAL INFILE '".$file_path."' INTO TABLE tmp_tbl";
DB::select($query);

Got following error

SQLSTATE[HY000]: General error: 2014 Cannot execute queries while other unbuffered queries are active. Consider using PDOStatement::fetchAll(). Alternatively, if your code is only ever going to run against mysql, you may enable query buffering by setting the PDO::MYSQL_ATTR_USE_BUFFERED_QUERY attribute. (SQL: LOAD DATA LOCAL INFILE 'products.csv' INTO TABLE tmp_tbl)

Which method I should continue using and why I am getting all these errors? I want to process all records in less time without any error.

Answer 1

try adding this: DB::connection()->disableQueryLog(); to the top of your script - your memory consumption may be coming from storing queries in memory.