Need to register 20000 scheduled lambda

Question

each day I need to dynamically schedule a lot of lambda, around 20K, each one on a specific timestamp for each lambda, during a day, each of this lambda will also need to schedule another lambda, on a specific timestamp for each lambda, this about 5 times (so maybe more than 20K in parallel during the day). I already saw how to use cloudwatch / eventbridge but "Each event bus in your account can have up to 100 EventBridge rules associated with it" and "Scheduled rules are supported only on the default event bus", so I understand I can't have more than a total of 100 schedule. Any idea how to do that? I am reading about stepfunction, "batch" but not sure about the reliability/stability and the final bill...

Thanks

Answer 1

I would highly recommend you to use Step Function .

Billing

You are charged based on the number of state transitions required to execute your application. If you go for Standard Workflow you will be charged $0.025 per 1,000 state transitions .

If you go for Express Workflows you will be charged $1.00 per 1M requests .

Though please remember, first 4,000 state transitions is free but need to be used within 12 months.

Reliability

State Functions are way reliable, you have much control over it. We all know, we can handle Exceptions in our lambda and do retries, you need not to worry if you forgot to handle Exceptions and retires, you can orchestrate this in State Functions .

Along with that you have a live visual monitor , where you can track the status of state run.

Few other things like, it is HIPPA compliant and you can track History of each execution .

To design State Machine you do not need to know any programming language too, knowing Json is good enough.

You can start designing you state machine from here or here .

Still you got any doubt, you can refer faqs .

Answer 2

As an alternative to scheduling 20,000 Lambda functions, and having them schedule another 20,000 functions, I would recommend taking a database approach :

• Use a database (your choice) to store the desired schedule, with one row per 'trigger' time
• Once per minute, trigger a single Lambda function that queries the database, retrieves a list of functions that should be triggered in that minute, then have it invoke those functions

So, you're effectively writing a scheduler for the functions.