node.js - stream http response from multiple calls

Question

In my express app, I want to make multiple calls to an API and stream each response back to the client as and when I receive it, instead of waiting for all of them.

For example, if I make requests to yelp for restaurants in San Francisco, Berkeley and Palo Alto in parallel, I shouldn't have to wait for all the responses to come back and be able stream them as they're available. How would I do this?

Answer 1

Since browsers wait till the entire response is received before passing the result to javascript, this isn't directly possible. You could, on the other hand, do it with websockets.

Possible architecture:

1. Server : Establish websocket connection with client
2. Server : Fire off 4 requests to APIs in parallel
3. Server : As data arrives on each connection, pass each packet to the websocket with something like {api: "yelp", data: ... }
4. Client : Keep appending incoming data to strings representing each api's response.
5. Server : When a connection is done, send a done message { api: "yepl", done: true }
6. Client : Upon receiving a done message, you have a full response from that API.

I highly doubt this is a good idea . It's far more complex and you'd be better off using 4 parallel requests from the client, or if possible, querying the apis directly from the browser.