Pg-promise inserts/transactions not working within async queue

Question

I have found a lot of things related to the use of pg-promise and await/async but nothing that quite answers my issue with async (the node/npm package) and in particular the interaction between async.queue and pg-promise queries.

My issue: I need to make a few millions computations (matching score) asynchronously and commit their results in the same async process in a postgres db. My main process is a promise that first computes all of the possible distinct combinations of two records from a table and segments them in chunks of a thousand pairs at a time.

These chunks of a thousand pairs (ie [[0,1], [0,2], ... , [0, 1000]] is my array of chunks' first index' content) are fed to an instance of async.queue that performs first the computation of the matching score then the db recording.

The part that has had me scratching my head for hours is that the db committing doesn't work whether it is using insert statements or transactions. I know for sure the functions I use for the db part work since I've written manual tests using them.

My main code is as follows:

'use strict';
const promise     = require('bluebird');
const initOptions = {
    promiseLib: promise
};
const pgp         = require('pg-promise')(initOptions);
const cn          = {connexion parameters...};
const db          = pgp(cn);
const async       = require('async');

var mainPromise = (db, php, param) => {

    return new Promise(resolve => {
        //some code computing the chunksArray using param
        ...
        var q = async.queue((chunk, done) => {

            var scores = performScoresCalculations(chunk);
            //scores is an array containing the 1000 scores for any chunk of a 1000 pairs
            performDbCommitting(db, scores);
            //commit those scores to the db using pg-promise
            done();

        }, 10);
        q.drain = () => {
            resolve(arr);
            //admittedly not quite sure about that part, haven't used async.queue much so far
        }
        q.push(chunksArray);
    )}.catch(err => {
        console.error(err);
    });

};

Now my scores array looks like this:

[{column1: 'value1_0', column2: 'value2_0', ..., columnN: 'valueN_0'}, ... , {column1: 'value1_999', column2: 'value2_999', column3: 'value3_999'}] with a thousand records in it.

My performDbCommitting function is as follows:

var performDbCommitting = (db, pgp, scores) => {

    console.log('test1');
    //displays 'test1', as expected
    var query = pgp.helpers.insert(scores, ['column1', 'column2', 'column3'], 'myScoreTable');
    console.log(query);
    //display the full content of the query, as expected
    db.any(query).then(data => {
        console.log('test2');
        //nothing is displayed
        console.log(data);
        //nothing is displayed
        return;
    }).catch(err => {
        console.error(err);
    });

}

So here is my problem:

• when testing "manually" performDbCommitting works perfectly, I've even tried a version with transactions, same works flawlessly,
• when used within async.queue everything in performDbCommitting seems to work until the db.any(query) call, as evidenced by the console.log displaying the info correctly until that point,
• no error is thrown up, the computations over chunksArray keep on going by groups of 1000 as expected,
• if I inspect any of the arrays (chunk, chunksArray, scores, etc) everything is as should be, the lengths are correct, their contents too.

pg-promise just doesn't seem to want to push my 1000 records at a time in the database when used with async.queue and that's where I'm stuck. I have no trouble imagining the fault lies with me, it's about the first time I'm using async.queue, especially mixed with bluebird promising and pg-promise.

Thank you very much in advance for taking the time to read this and shed any light on this issue if you can.

Answer 1

I was experiencing this same issue on one of my machines in particular but none of the others. What worked for me was updating pg-promise from version 10.5.0 to version 10.5.6 (via npm update pg-promise).

Answer 2

Your mainPromise doesn't wait for performDBCommitting to finish:

should be like:

//commit those scores to the db using pg-promise
performDbCommitting(db, scores).then(()=>{done();});

and performDBCommitting needs to return the promise too:

return db.any(query).then(data => {
    console.log('test2');
    //nothing is displayed
    console.log(data);
    //nothing is displayed
    return null;
}).catch(err => {
    console.error(err);
    return null;
});