Node.JS- R, Python heavy computation identify when a callback returns and store that result

Question

I expose a Node.js REST API that below call R and Python scripts that do heavy computations. Before I call the R and Python computation i need to identify the callback and store the specific instance of this callback assign an unique id and immediately send a fast response with the id of the job o the consumer. In future, the consumer will query the state of the job/callback with the id.

What is the best approach to manage this? How to identify a callback and store a unique id of it in a database?

Answer 1

One way to do it it to use some distributed task framework, like Celery . Once you get the request, you start the heavy computations in a task, use the task's uuid as an identifier, and send it back to the client with a 202 Accepted response and a link that he can follow to your API to find the results.

Inside the task call, you create an empty entity that signals somehow it's still under processing, acessible under the URI in the link you sent to the client. When the task is finish, you save the results.