MYSQL configuration for multiple and bulk records insert query

Question

I am inserting more then 1 million records and my each insert query inserts 15000 records by single query at a time it is taking 1 min to inserting 1 million records

like this :-

insert into records (data) values ('a'),('b'),('c'),('d')........................

but when i execute 2 query at a time mysql goes slow and it is taking 25-30 min.

what mysql configuration i should use for solve this problem

Answer 1

Are you saying you insert 15000 records like this:

insert into records (data) values ('a'),('b'),('c'),('d')
insert into records (data) values ('a'),('b'),('c'),('d')
insert into records (data) values ('a'),('b'),('c'),('d')
insert into records (data) values ('a'),('b'),('c'),('d')
insert into records (data) values ('a'),('b'),('c'),('d')
insert into records (data) values ('a'),('b'),('c'),('d')
insert into records (data) values ('a'),('b'),('c'),('d')
insert into records (data) values ('a'),('b'),('c'),('d')

.....

If so try using INSERT ALL

INSERT ALL INTO records (data) values
('a'),('b'),('c'),('d')
('a'),('b'),('c'),('d')
('a'),('b'),('c'),('d')
('a'),('b'),('c'),('d')
('a'),('b'),('c'),('d')
('a'),('b'),('c'),('d')
('a'),('b'),('c'),('d')
('a'),('b'),('c'),('d')
('a'),('b'),('c'),('d')

...........

It may take some work but if you group all of the inserts that go to the same table so you can use INSERT ALL you can reduce the number of statements the server needs to parse, which is very expensive.

Answer 2

You should use transactions . This will improve but not necessarily fix your issue.

In PHP:

$pdo = new PDO($dsn, $username, $password);
$pdo->beginTransaction();

foreach ($somethings as $something) {
    $stmt = $pdo->prepare("INSERT INTO records VALUES (?)");
    $stmt->execute([$something]); // PHP 5.5 will like this
    $stmt->execute(Array($something)); // PHP 5.4 or under.   
}

$pdo->commit();

But really for such large data entries you should consider just generating queries which do send 15000 values at once...