Is parallelization of network requests a good use for Node.js workers?

Question

I'm experimenting with the Node.js worker_threads module, with the aim of 'collecting' the result of many requests to different APIs. Is this a good use case for Workers?

For example:

import { Worker } from 'worker_threads'

const API_ADDRESSES = [... maybe 20 different URIs]

const results = await Promise.allSettled(
  API_ADDRESSES.map(
    uri => new Promise(
      (resolve, reject) => {
        const worker = new Worker(... filepath.js, { workerData })
        worker.on('message', resolve)
        worker.on('error', reject)
      }
    )
  )
)

// The Worker then uses axios/node-fetch/etc to make a network request and returns data as a message

If this is NOT a good use case for workers, what would be a better approach? Also, if not a great idea, why is this NOT a good use case for workers?

Having tried this, it seems to work fine but I don't really know how to assess it from a performance perspective.

==== EDIT

The reason that I thought to try this approach instead of

await Promise.allSettled(API_ADDRESSES.map(uri => fetch(uri, {....})))

is that per result I might want to process the response before returning it (ie a result might be a lot of numbers that I want a correlation coefficient for).

Answer 1

Basically, http request in Node.js isn't a blocking operation (it's mostly true). so using a worker here is redundant and not recommended.

what would be a better approach

This can change from case to case, but this should suffice for most cases:

await Promise.all([fetch(...), fetch(...),...]);

more information can be found here .

Answer 2

Well, This seems to work fine as you want to get your calls done in different threads. But as NodeJS is known to have async IO already. So, to do a network job a simple request library usage will suffice.

worker_threads on the other hand are heavier as to incorporate CPU usage use cases. And worker threads communicate with each other with IPC calls as they behave as a completely separate process spawn.

Some use cases for worker_threads would be

1. Create a cluster of http_servers.
2. assigning a CPU intensive job on a new thread. etc...

Answer 3

Is parallelization of network requests a good use for Node.js workers?

Not really. Node already parallelizes network requests.

Is parallelization of CPU-heavy processing of network requests a good use for Node.js workers?

Probably. If the processing of the network request is going to take any significant amount of time, parallelizing that processing can result in performance benefits. You can determine this with certainty by benchmarking both approaches.

Just note that the key operator here is that you are parallelizing the cpu-heavy processing . The network request portion itself is already parallelized quite effectively.