Inserting many rows to mysql using node.js

Question

I'm trying to import a large file into a database. After many failed tries to import as csv file through mySQL, I decided to create a small node script that can read the file and insert the records one by one.

I've got about 10 differently formatted files of 80mb each. The current script is for one file that has an id on each line and nothing more (this particular table consists only of an id field and a status field), and this is its current state:

var mysql      = require('mysql');
var connection = mysql.createConnection({
    host     : 'hostname',
    user     : 'username',
    password : 'password',
    database : 'database'
});

var rl = require('readline').createInterface({
    input: require('fs').createReadStream('fileToRead.txt')
});

connection.connect();
rl.on('line', function (line) {
    var query = 'REPLACE INTO database.tablename(field1,field2) VALUES ("'+line+'",0);';
    connection.query(query, function(err) {
        if (err) {
            console.log("ERR:"+err);
            connection.end();
        }
    });
});

It works fine for about ten to twelve lines, and then throws the following mistake:

   <--- Last few GCs --->

   51338 ms: Scavenge 699.0 (738.6) -> 699.0 (738.6) MB, 8.7 / 0 ms (+ 15.0 ms i
n 1 steps since last GC) [allocation failure] [incremental marking delaying mark
-sweep].
   53709 ms: Mark-sweep 699.0 (738.6) -> 698.9 (738.6) MB, 2360.5 / 0 ms (+ 15.0
 ms in 2 steps since start of marking, biggest step 15.0 ms) [last resort gc].
   56065 ms: Mark-sweep 698.9 (738.6) -> 698.9 (738.6) MB, 2360.2 / 0 ms [last r
esort gc].



    <--- JS stacktrace --->
==== JS stack trace =========================================

Security context: 1DF25599 <JS Object>
    1: emit [events.js:~117] [pc=23C30364] (this=1027D245 <a Protocol with map 3
2339A39>,type=1DF4D5B1 <String[7]: enqueue>)
    2: arguments adaptor frame: 2->1
    3: _enqueue [path\node_modules\mysql\lib\protocol\
Protocol.js:~128] [pc=107BD3D8] (this=1027D245 <a Protocol with map 32339A39>,se
quence=157A3225 <a Query with map 3233C379>)
    4: /* anonymous */ [path...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

I'm not used to node, so I don't really know what that means. I believe it has to do with the query being called from inside a loop that moves faster than the query, but I'm not sure about it and wouldn't know how to handle it if that was the case.

Any help is appreciated.

Sorry if my english fails.

Answer 1

The problem is that you are ending the connection to the database after the first query ends, so, only the amount of querys that had enough time to go to the database should be executed and my guess is that only the first one will be inserted (maybe I am wrong here).

Since you maybe are doing a " one time only script " of node, you could fix your problem just removing that line, something like this:

var mysql      = require('mysql');
var connection = mysql.createConnection({
    host     : 'hostname',
    user     : 'username',
    password : 'password',
    database : 'database'
});

var rl = require('readline').createInterface({
    input: require('fs').createReadStream('fileToRead.txt')
});

connection.connect();
rl.on('line', function (line) {
    var query = 'REPLACE INTO database.tablename(field1,field2) VALUES ("'+line+'",0);';
    connection.query(query, function(err) {
        if (err) {
            console.log("ERR:"+err);
            //I am not closing the connection anymore
        }
    });
});

If your scripts will be useful for many times (once a month, once a day or something like that) I will sugest a better solution, maybe using async and a pool of connections.