Implementing Lock in Node.js

Question

Preface:

In order to solve my problem, you have to have knowledge in the following areas: thread-safety, Promise, async-await.

For people who are not familiar with TypeScript, it's just normal JavaScript (ES6) with type annotations.

---

I have a function named excludeItems that accepts an item list (each item is a string), and calls an API (that excludes the item) for each item. It's important not to call the API twice for the same item, not even in different executions of the function, so I save in a local DB the items that are already excluded.

async function excludeItems(items: string[]) {
            var excludedItems = await db.getExcludedItems();

            for (var i in items) {
                var item = items[i];
                var isAlreadyExcluded = excludedItems.find(excludedItem => excludedItem == item);

                if (isAlreadyExcluded) {
                    continue;
                }

                await someApi.excludeItem(item);
                await db.addExcludedItem(item);
            }
     }

This function is called asynchronously by its client several times instantaneously, meaning the client calls the function say 5 times before the first execution is completed.

A concrete scenario:

excludeItems([]);
excludeItems(['A','B','C']);
excludeItems([]);
excludeItems(['A','C']);
excludeItems([]);

In this case, although Node.js is single-threaded, the Critical Section problem is existing here, and I get the wrong results. This is my "excludedItems" collection in my local DB after the execution of that scenario:

[{item: 'A'},
{item: 'B'},
{item: 'C'},
{item: 'A'},
{item: 'C'}]

As you can see, the last 'A' and 'C' are redundant (meaning that the API was also called twice for these items).

It occurs due to the await statements in the code. Every time an await statement is reached, a new Promise is created under the hood, therefore although Node.js is single-threaded, the next async function that was waiting to be executed is getting executed, and that way this critical section is executed parallelly.

To solve that problem, I've implemented a locking mechanism:

var excludedItemsLocker = false;
async function safeExcludeItems(items: string[]) {
        while (excludedItemsLocker) {
            await sleep(100);
        }
    try {
        excludedItemsLocker = true;

        var excludedItems: string[] = await db.getExcludedItems();

        for (var i in items) {
            var item = items[i];
            var isAlreadyExcluded = excludedItems.find(excludedItem => excludedItem == item);

            if (isAlreadyExcluded) {
                continue;
            }

            await someApi.excludeItem(item);
            await db.addExcludedItem(item);
        }
    }
    finally {
        excludedItemsLocker = false;
    }
}

async function sleep(duration: number): Promise<Object> {
    return new Promise(function (resolve) {
        setTimeout(resolve, duration);
    });
}

However, this implementation does not work for some reason. I still get more than one (alleged) "thread" in the critical section, meaning it's still getting executed parallelly and my local DB is filled with the same wrong results. BTW the sleep method works as expected, its purpose is just to give CPU time to the next function call that's waiting to be executed.

Does anybody see what's broken in my implementation?

BTW I know that I can achieve the same goal without implementing a Lock, for example by calling to db.getExcludedItems inside the loop, but I want to know why my Lock implementation is broken.

Answer 1

If the parameters are:

['A','B','C']

and db.getExcludedItems() returns:

[{item: 'A'},
{item: 'B'},
{item: 'C'}]

Then you are trying to find a string in an array of objects, which will always return undefined:

var isAlreadyExcluded = excludedItems.find(excludedItem => excludedItem == item);

Just a thought, because I can't see any problem with the locking itself, it should work as expected.

Answer 2

I encountered a similar problem a while back and I ended up implementing a ticket system where each "thread" would request a ticket and wait in a queue (I know it's not a thread, but it's easier to say than 'next set of functions in the event loop'<\/em> ). It's an NPM package found at promise-ticket<\/code><\/a> , but the crux of the solution was to have a generator function returning promises that would resolve when an EventEmitter<\/code> would emit it's ticket number

let emitter = new EventEmitter();
let nextTicket = 0;
let currentTicket = 0;
let skips = [];

const promFn = (resolve) => {
    let num = currentTicket++;
    emitter.once(num, () => resolve(num));
};
const generator = (function* () {
    while(true) yield new Promise(promFn);
})();

// Someone takes a ticket from the machine
this.queue = (resolveValue) => {
    let ticketNumber = currentTicket;
    let p = generator.next().value;
    if(resolveValue !== undefined) p = p.then(() => resolveValue);
    if(skips.includes(ticketNumber)) emitter.emit(ticketNumber);
    return p;
};