Knex bulk insert not waiting for it to finish before passing to the next async operation

Question

I am having a problem where I am making a bulk insert of multiple elements into a table, then I immediatly get the last X elements from that table that were recently inserted but when I do that it seems that the elements have not yet been inserted fully even thought I am using async await to wait for the async operations.

I am making a bulk insert like

const createElements = elementsArray => {
    return knex
        .insert(elementsArray)
        .into('elements');
};

Then I have a method to immediately access those X elements that were inserted:

const getLastXInsertedElements = (userId, length, columns=['*']) => {
    return knex.select(...columns)
            .from('elements').where('userId', userId)
            .orderBy('createdAt', 'desc')
            .limit(length);
}

And finally after getting those elements I get their ids and save them into another table that makes use of element_id of those recently added elements.

so I have something like:

// A simple helper function that handles promises easily
const handleResponse = (promise, message) => {
    return promise
        .then(data => ([data, undefined]))
        .catch(error => {
            if (message) {
                throw new Error(`${message}: ${error}`);
            } else {
                return Promise.resolve([undefined, `${message}: ${error}`])
            }
        }
    );
};

async function service() {
    await handleResponse(createElements(list), 'error text'); // insert x elements from the list
    const [elements] = await handleResponse(getLastXInsertedElements(userId, list.length), 'error text') // get last x elements that were recently added

    await handleResponse(useElementsIdAsForeignKey(listMakingUseOfElementsIds), 'error text'); // Here we use the ids of the elements we got from the last query, but we are not getting them properly for some reason
}

So the problem: Some times when I execute getLastXInsertedElements it seems that the elements are not yet finished inserting, even thought I am waiting with async/await for it, any ideas why this is? maybe something related to bulk inserts that I don't know of? an important note, all the elements always properly inserted into the table at some point, it just seems like this point is not respected by the promise (async operation that returns success for the knex.insert).

Update 1: I have tried putting the select after the insert inside a setTimeout of 5 seconds for testing purposes, but the problem seems to persist, that is really weird, seems one would think 5 seconds is enough between the insert and the select to get all the data.

I would like to have all X elements that were just inserted accessible in the select query from getLastXInsertedElements consistently.

Answer 1

Which DB are you using, how big list of data are you inserting? You could also test if you are inserting and getLastXInsertedElements in a transaction if that hides your problem.

Doing those operations in transaction also forces knex to use the same connection for both queries so it might lead to a tracks where is this coming from.

Another trick to force queries to use the same connection would be to set pool's min and max configuration to be 1 (just for testing is parallelism is indeed the problem here).

Also since you have not provided complete reproduction code for this, I'm suspecting there is something else here in the mix which causes this odd behavior. Usually (but not always) this kind of weird cases that shouldn't happen are caused by user error in elsewhere using the library.

I'll update the answer if there is more information provided. Complete reproduction code would be the most important piece of information.

Answer 2

I am not 100% sure but I guess the knex functions do not return promise by default (but a builder object for the query). That builder has a function called then that transforms the builder into a promise. So you may try to add a call to that:

... 
limit(length)
.then(x => x); // required to transform to promise

Maybe try debugging the actual type of the returned value. It might happen that this still is not a promise. In this case you may not use async await but need to use the then Syntax because it might not be real js promises but their own implementation.

Also see this issue about standard js promise in knex https://github.com/knex/knex/issues/1588

Answer 3

In theory, it should work.

You say "it seems"... a more clear problem explanation could be helpful.

I can argue the problem is that you have elements.length = list.length - n where n > 0 ; in your code there are no details about userId property in your list ; a possible source of the problem could be that some elements in your list has a no properly set userId property.