Unexpected return of async/ await function

Question

The scraper I am making has a register route to start the scraping process. Since the scraping process is very long, I decided to create another route that my front-end will make a request to every 15 seconds to get the data. The problem I am encountering is that the route to get the data is returning an empty object even when the scraping is done. In reality, it should have returned the scraped data and NOT have returned a 404 error (refer to code below). I have included extra code in case it is the cause of the problem.

const scraper = async (email, password) => {
  let grades = [[], [], [], []];
  let category_weights = [];
  try {
    // try to scrape website
    // this block of code contains await functions
  } catch (e) {
    console.log(e);
    return;
  } finally {
    return {
      grades,
      category_weights
    };
  }
};

let result;

app.post("/api/register", (req, res) => {
  result = scraper(req.body.email, req.body.password);
  res.sendStatus(200);
});

app.get("/api/data", (req, res) => {
  console.log(result["grades"]); // undefined
  console.log(JSON.stringify(result, null, 4)); // {}
  result.hasOwnProperty("grades") ? res.json(result) : res.sendStatus(404);
});

Answer 1

The code is returning 404 status code because result is a Promise , and it does not have grades property.

You have to wait until the promise is done. You can use await for that. That way, once the scraper is done, calls to /api/data will return the data, instead of a 404 status code.

In order to avoid the timeout issue, you can either increase the timeout for that specific route:

req.setTimeout(1000 * 60 * 5); // or whatever value you want

Or just end the request as soon as /api/register is hit, and then run the scraper function, awaiting the result.

app.post("/api/register", async(req, res) => {

    res.sendStatus(200);

    try {
        result = await scraper(req.body.email, req.body.password);

    } catch(e) {
        console.log('Scraper failed');
    }

});

app.get("/api/data", (req, res) => {
  console.log(result["grades"]); // undefined
  console.log(JSON.stringify(result, null, 4)); // {}
  result.hasOwnProperty("grades") ? res.json(result) : res.sendStatus(404);
});

And here's proof that JSON.stringify on a Promise will return {}

 console.log(JSON.stringify(new Promise(() => {}), null, 4)); // {}