Invoke a AWS Step functions by API Gateway and wait for the execution results

Question

Is it possible to invoke a AWS Step function by API Gateway endpoint and listen for the response (Until the workflow completes and return the results from end step)?

Currently I was able to find from the documentation that step functions are asynchronous by nature and has a final callback at the end. I have the need for the API invocation response getting the end results from step function flow without polling.

Answer 1

I guess that's not possible.

It's async and also there's the API Gateway Timeout

You don't need get the results by polling, you can combine Lambda, Step Functions, SNS and Websockets to get your results real time.

If you want to push a notification to a client (web browser) and you don't want to manage your own infra structure (scaling socket servers and etc) you could use AWS IOT. This tutorial may help you to get started:

http://gettechtalent.com/blog/tutorial-real-time-frontend-updates-with-react-serverless-and-websockets-on-aws-iot.html

If you only need to send the result to a backend (a web service endpoint for example), SNS should be fine.

Answer 2

This will probably work: create an HTTP "gateway" server that dispatches requests to your Steps workflow, then holds onto the request object until it receives a notification that allows it to send a response.

The gateway server will need to add a correlation ID to the payload, and the step workflow will need to carry that through.

One plausible way to receive the notification is with SQS.

Some psuedocode that's vaguely Node/Express flavoured:

const cache = new Cache(); // pick your favourite cache library
const gatewayId = guid(); // this lets us scale horizontally
const subscription = subscribeToQueue({ 
    filter: { gatewayId },
    topic: topicName,
});

httpServer.post( (req, res) => {
   const correlationId = guid();
   cache.add(correlationId, res);
   submitToStepWorkflow(gatewayId, correlationId, req);
});

subscription.onNewMessage( message => {
   const req = cache.pop(message.attributes.correlationId);
   req.send(extractResponse(message));
   req.end();
});

(The hypothetical queue reading API here is completely unlike aws-sdk's SQS API, but you get the idea)

So at the end of your step workflow, you just need to publish a message to SQS (perhaps via SNS) ensuring that the correlationId and gatewayId are preserved.

To handle failure, and avoid the cache filling with orphaned request objects, you'd probably want to set an expiry time on the cache, and handle expiry events:

cache.onExpiry( (key, req) => {
   req.status(502);
   req.send(gatewayTimeoutMessage());
   req.end();
}

This whole approach only makes sense for workflows that you expect to normally complete in the sort of times that fit in a browser and proxy timeouts, of course.