async/await functions have strange behaviour

Question

I have a nodejs / javascript app and have some issues with async / await. I have checked many of the suggestions given here, but cannot find one that fits my issue. If there is one out there I apologise in advance.

I have a function that gets information from a websocket stream. This information is buffered, awaiting further processing.

The function that processes the information from the stream (updateOrderBook) is called every second, using a setTimeout.

In order to be able to process the stream data, an order book must exist. If it doesn't one shall be created. That function (createOrderBook) needs to get data from a REST API and here is where I have the problems.

In updateOrderBook I'm awaiting the createOrderBook function. createOrderBook in its turn awaits axios from returning the REST API data. But whilst awaiting this API data it seems that control is returned back to updateOrderBook.

So what am I overlooking here? and more importantly how can I fix it?

const updateOrderBook = () => {
  // select the coin at hand
  dataBuffer.forEach(async (data, index) => {
    dataBuffer.splice(index, 1);
    let symbol = data.s.toLowerCase();

    if (orderBook.filter(orders => orders.coin === symbol).length === 0) {
      await createOrderBook(symbol);
      console.log(`${symbol}: Orderbook created`);
    }

    if (typeof lastUpdateId[symbol] == "undefined") {
      console.log(
        symbol,
        "UNDEFINED",
        orderBook.filter(orders => orders.coin === symbol).length === 0
      );
    }
    ...
  setTimeout(() => updateOrderBook(), 1000);
};

const createOrderBook = async (symbol) => {
  console.log(symbol, "ENTERED COB");
  orderBook.push({ coin: symbol, bids: [], asks: [] });
  firstEvent[symbol] = true;
  try {
    const response = await axios.get(
      `https://www.binance.com/api/v3/depth?symbol=${symbol.toUpperCase()}&limit=1000`
    );
    console.log(symbol, "BEFORE ENTERING HIO");
    handleInitialOrderBook(response, symbol.toLowerCase());
    console.log(symbol, "COB");
  } catch (error) {
    console.log(symbol, error);
  }
};

Expected log:

xrpusdt BEFORE ENTERING HIO
xrpusdt HIO
xrpusdt COB
xrpusdt: Orderbook created

Actual log:

xrpusdt ENTERED COB
xrpusdt UNDEFINED false
xrpusdt BEFORE ENTERING HIO
xrpusdt HIO
xrpusdt COB
xrpusdt: Orderbook created

As you can see the UNDEFINED log which I expect to happen, if at all, after createOrderBook finishes came before createOrderBook completely finished

Answer 1

The problem lies within:

dataBuffer.forEach(async (data, index) => {
  // ...
});

Although the callback function is an async function. forEach doesn't check the return value of this callback and does not await the returned promise before calling the next iteration.

Currently the first element in the buffer is waiting for createOrderBook to finish, while the second element skips the if-block:

if (orderBook.filter(orders => orders.coin === symbol).length === 0)

This is because createOrderBook calls orderBook.push(...) before awaiting, thus the element is already pushed and the JavaScript engine will move on the the next element in forEach as soon as it needs to await within createOrderBook .

So how do you solve this? The easiest way would be to not use a forEach call, but instead use for...of . Because you are not creating new callback functions the main function will await the promise and only move on once it is resolved.

for (const [index, data] of dataBuffer.entries()) {
  // ...
}

For this to work you do have to change updateOrderBook into an async function. The code block above assumes that dataBuffer is an array, if it is another object you might have to check the documentation on how to get an iterator with both the index and element.

Note: By using dataBuffer.splice(index, 1) within the loop you get some strange behaviour. On the first element index is 0 and the first element is removed. This will shift the whole array. Normally you don't want to mutate an array while you are iterating over it. This can be solved in one of two ways:

1. Use for (const data of dataBuffer) instead and splice of all elements after you are done iterating. dataBuffer.slice(0) (clears the array).

2. Use a while-loop instead.

    while (dataBuffer.length > 0) { const data = dataBuffer.shift(); // ... } 

 const letters = ["a", "b", "c", "d", "e", "f", "g"]; for (const [index, letter] of letters.entries()) { console.log(index, letter); letters.splice(index, 1); } console.log(letters);

---

const updateOrderBook = async () => {
  // select the coin at hand
  while (dataBuffer.length > 0) {
    const data = dataBuffer.shift();
    let symbol = data.s.toLowerCase();

    if (orderBook.filter(orders => orders.coin === symbol).length === 0) {
      await createOrderBook(symbol);
      console.log(`${symbol}: Orderbook created`);
    }

    if (typeof lastUpdateId[symbol] == "undefined") {
      console.log(
        symbol,
        "UNDEFINED",
        orderBook.filter(orders => orders.coin === symbol).length === 0
      );
    }
  }

  // ...
  setTimeout(() => updateOrderBook(), 1000);
};