Why is my AWS Lambda node.js mysql query not returning?

Question

I'm trying to write some external data into some local tables. We'll be looping through an array, writing most of the data in each array element to the main table and the rest to related tables, replacing all the data each time.

I've stripped the code down to the bare bones to show the problem I'm having. The DELETE runs fine, but the INSERT runs only once, and doesn't even return.

I have a screenshot of the output at https://imgur.com/a/zA6Hz8g . In it, you can see that the code for the DELETE runs fine (ComQueryPacket sent, OkPacket returned) but when it gets to the INSERT, the ComQueryPacket is sent but nothing is returned. And then the code just falls through.

This results in the first row writing successfully, but no subsequent rows get written.

I've tried changing the connection to use pools, but that didn't help either.

Any ideas?

var mysql = require('mysql');
var promise = require('promise');

const con = mysql.createConnection({
    <connectionInfo>,
    debug: true
});

function connectToDB() {
    return new promise((resolve, reject) => {
        console.log("IN connectToDB");
        con.connect(function(err) {
            if (err) {
                console.log("ERROR: Could not connect -- " + err);
                reject;
            }
            console.log("Connected!");
            resolve();
        });
    });
}

function deleteExistingMainRow() {
    return new promise((resolve, reject) => {
        var query = "DELETE FROM my_table";
        con.query(query, [],
            function(err, result) {
                if (err) {
                    console.log("ERROR in deleteExistingMainRow: " + err);
                    reject;
                }
                else {
                    console.log("DEBUG: Successful delete of main row");
                    resolve();
                }
            });
    });
}

function writeMainRow(data_row) {
    return new promise((resolve, reject) => {
        console.log("IN writeMainRow");
        var query = 'INSERT INTO my_table SET id = ?';

        con.query(query, [data_row.id],
            function(err, result) {
                console.log("YES we tried to query");
                if (err) {
                    console.log("ERROR in writeMainRow: " + err);
                    reject(err);
                }
                else {
                    console.log("DEBUG: Successful write of main row");
                    resolve();
                }
            });
    });
}

exports.handler = function(event, context) {
    connectToDB().then(function(script) {
        deleteExistingMainRow().then(function(script) {
            var data = [{ "id": 1 }, { "id": 2 }, { "id": 3 }];
            data.forEach(data_row => {
                writeMainRow(data_row).then(function(script) {
                        console.log("DEBUG: Main row written in forEach");
                    },
                    function(err) {
                        if (err) { console.log("ERR"); } process.exit(0);
                    }());
            });
            console.log("DEBUG: Hey we're exiting now");
            con.commit;
            con.end(function(err) {
                console.log("Error on con end: " + err);
            });
            context.done(null, "DONE");
            process.exit(0);
        });
});

};

Answer 1

The reason why your code is not behaving as you expect is because of the asynchronous nature of NodeJS.

Your for_each loop spawns several threads that are going to INSERT the data in your database.
As soon as these threads are started, the rest of the code will execute, starting with console.log("DEBUG: Hey we're exiting now");

So the commit happens before all the INSERT calls are done and, more importantly, you're calling Process.exit() in your code. This terminates the runtime, even before the INSERT can finish. Call callback() instead as per https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html

Handling your multiple asynchronous writes can be done differently. First, as grynets commented before me, I would strongly suggest to rewrite your code using async / await to make the call easier to read. Then, you have to understand that each call to writeMainRow will return its own Promise and your code must wait for ALL promises to complete before to commit() and to callback()

Promise.all(...) will do that for you. See the doc at https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

Answer 2

Just a few moths ago AWS made Node.js v 8.10 runtime available in lambda.
Which means, you can use async/await and Promises .
So, we can rearrange code to something like this:

exports.handler = async (event, context) => {
     const dbConnection = await connectToDB();

     await deleteExistingMainRow();

     const data = [{ "id": 1 }, { "id": 2 }, { "id": 3 }];

     // use here for...of loop to keep working with async/await behaviour
     for(const data_row of data){
       await writeMainRow(data_row);
     }
}

Also, you can rewrite your code to use native Promises or async/await functions.
And of course, cover logic on try/catch block, I've skipped them for simplicity.

Answer 3

try using INSERT INTO table_name(id) VALUES (?); I know both your query and the above query works the same. Just give it a Try.

And just make sure your for loop is working properly sending values to the writeMainRow(function). It wouldnt show an error you if pass an empty value and make sure you are not passing the same values in the for loop. And i think you have to pass writeMainRow(data_row.id) rather than writeMainRow(data_row) .

Hope this helps. And one more suggestion if you are updating multiple rows there are options in mysql node library like transactions. Using those functions will be more effective and you can roll back the result if you face error. Other option is to write procedures, in which case you mysql server will bear the computation.