AWS Step Function Map task

Question

I am thinking of using this solution to process about 3000 objects from S3.

https://docs.aws.amazon.com/step-functions/latest/dg/sample-map-state.html

My use case is that I need to loop through all items one at a time, passing each file to multiple lambda functions for processing, until all items have been processed. I haven't been able to find any information to tell me if there are limitations to this approach. All the examples I have seen use a small number of items and I was wondering if there was any reason that I shouldn't use it for 3000 (or more) items.

Answer 1

AWS Step Functions has 25,000 events limit for maximum execution history size. So if you have 5 steps for processing one file you can't process more than 5000 items.

To overcome this limit:

• Wrap all the process for the file in one step functions and call if from your parent step functions (Nested Step Functions).

• Instead of passing 1 file to your map or nested step functions, pass a batch of them and them iterate over them in your child state machine. So for example if your batch 10 files, instead of 5000 you can process 50000.

• The other workaround is to start another Execution when you reached the 25000 limits, but solving it with the first 2 options is easier.