ASP.NET Web API - Multiple Long-Running API calls best practices?

Question

I'm looking for best practices to implement the scenario below - I have some solutions for that already (not implemented) but wanted to make sure of it that's why I'm sharing the situation here.

Scenario

I have an ASP.NET Web API (netcore) application which is supposed to return back a complex data structure formatted in json . Let's call this WebAPI.NumberOne . This API internally needs to call out for another API endpoint which does the main job, meaning that, calling out 22 external APIs using async await pattern having Task.WaitAll in place to make sure all the external APIs have completed the job, And then, to do some processing on each response such as parsing the responses to the result type, and finally to return the result to WebAPI.NumberOne . I want to call the second one WebAPI.NumberTwo .

Restriction

We cannot simplify the WebAPI.NumberTwo at the moment and it has to do all those heavy lifting on its own. And also that, our client does not want to have any new technology introduced to the project such as any message queue platforms.

Problem:

I have this concern of receiving timeout from WebAPI.NumberOne as the other API takes sometimes like 10 minutes to complete the job. And probably some other potential risks that I don't know of now.

My thoughts

1. I was thinking about implementing a Web Hook Receiver and Listener so that when WebAPI.NumberTwo completes its job to POST the results to the receiver and on the other hand the listener to notify the WebAPI.NumberOne about the results.
2. I believe using Service Bus and Message Queues concept can also be quite helpful for this scenario but as our client does not want to have new technology in place at the moment I didn't go for any message queue mechanisms.

So, having the above said, what are some usual best practice for such scenarios? I am more keen to know of different approaches and best practice even regardless of the restrictions I mentioned above. Appreciate your time and help.

Answer 1

what we've done is implement queuing using rabbitmq and masstransit (this would be a new tech for your client though).

All the api calls do is queue a message and return 200 ok.

The message consumers then call the functions that handle the heavy lifting.

To integrate other api's we publish an event and those other api consumers can then process the message.

Given the tags on this post, it's possible you already have azure queues available to you or at least msmq.

With your webhook proposal what happens if there is an outage, or a single component needs an update? with queues the message just sits there until its consumed.