How to use await on https module to get response in node.js

Question

why does the 1st log print response data, and 2nd log prints function definition instead of response data. is there a way to debug and see the flow?

const https = require("https");
const substr = "spider";
const pageNum = 1;

let list = [];

const fetch = (url) =>
  https.get(url, async (res) => {
    res.setEncoding("utf-8");
    let ttt;
    await res.on("data", function (data) {
      ttt = data;
    });
    console.log(ttt); //1st log
    return ttt;
    
  });

  
((substr) => {
  totalPages = 1;

  pageNum;
  for (let i = 1; i <= totalPages; i++) {
    const res = fetch(
      "https://jsonmock.hackerrank.com/api/movies/search/?Title=" +
        substr +
        "&page=" +
        i
    );
    console.log(res);//2nd log
  }
})("s");

Answer 1

await is a keyword that is used in combination with a Promise value. It tells the code to wait until the Promise resolves a value, or tells it to throw if there is an error. If you check the function definition for fetch you will see that it returns a Promise instead of the response object directly. You need to use the await keyword to resolve that Promise.

const res = await fetch(...

Answer 2

You are not awaiting the fetch request in the loop. Res just holds the Promise that fetch returns. You instead need to have a try and catch block around the statement with an await before the fetch call or you simply use.then() and.catch().

Answer 3

There are some more reasons besides the missing await statement before the call to fetch. The reason is because the fetch function does not in fact return a promise.

You set it as an arrow function. Arrow functions will do an implicit return if the function body only has a single expression. So the fetch function returns the result of the https.get() function call, which is just the function signature as it works with the callback concept.

Additionally you did not handle the event listener correctly, as the res.on('data', <function here>) line does not return a promise, so the await keyword did not do anything.

Let's see how I would rewrite your code to run as expected:

const https = require('https')
const substr = 'spider'
const pageNum = 1

let list = []

// Create an async function fetch that takes the url
const fetch = (url) => {
  // Return a promise
  return new Promise((resolve, reject) => {
    // Call the https.get function
    https.get(url, (res) => {
      res.setEncoding('utf-8')
      // Set the event listener
      res.on('data', function (data) {
        // Here the actual data event is _happening_. Now we have the return value
        console.log(data) // 1st Log
        resolve(data)
      })
    })
  })
}

(async (substr) => {
  totalPages = 1;

  pageNum;
  for (let i = 1; i <= totalPages; i++) {
    // await the promise returned from fetch
    const res = await fetch(
      "https://jsonmock.hackerrank.com/api/movies/search/?Title=" +
      substr +
      "&page=" +
      i
    );
    console.log(res);//2nd log
  }
})("s");

I hope this attempt at an explanation helps!

Answer 4

found it. looks like https doesn't return anything, you can use Promise/resolve to return response after the request is completed.

const https = require("https");
const substr = "spider";
const pageNum = 1;

let list = [];


const fetch = async (url) => {
  return new Promise((resolve) => {
    https.get(url, async (res) => {
      res.setEncoding("utf-8");
      let ttt;
      await res.on("data", function (data) {
        ttt = data;
        resolve(data);
      });
      res = ttt;
      console.log(ttt);
    });
  });
};

(async (substr) => {
  totalPages = 1;

  pageNum;
  for (let i = 1; i <= totalPages; i++) {
    const res = await fetch(
      "https://jsonmock.hackerrank.com/api/movies/search/?Title=" +
        substr +
        "&page=" +
        i
    );
    console.log(res);
  }
})("s");