aws step function/state machine, can you launch and forget?

Question

considerations:

• I have a step that will probably take longer than a lambda can wait.
• the step involves a s3 copy (terabytes of data)
• I don't need to wait for the successful result

Do I need to wrap the state machine in something like launching it from a docker container? (because docker can wait longer than a lambda for the return value which I don't need) or can I launch this state machine from a lambda and not wait for the result.

Answer 1

Step Functions Standard Workflows executions can run for up to 1 year. The workflow execution will run asynchronously during that time and you can monitor the progress from the console, CLI, or SDK. This is completely managed and you don't need to run any processes, such as a Docker container, on your side to keep it going.

That said, Step Functions will orchestrate the actions over the course of that period but you will still need to implement the tasks that it will orchestrate. For a bulk S3 copy, it seems you may be running a docker container to do the actual copy in which case you could use the Service Integration with ECS (including Fargate) to make that easier.

Alternatively, you could the new Distributed Map capability to avoid needing to write any code in Lambda or Container if the source and destination bucket are in the same region. This is an example of a bulk delete that could be easily modified to do a copy.