Is there a way to programmatically know how many concurrent instances of a Lambda function are executing?

Question

Lets suppose, at a given time, several instances of a particular Lambda function have been invoked asynchronously,

Then, is there a way to find how many active concurrent instances of the Lambda function are currently running?

In the account level metrics in the Dashboard we can find the number of concurrent executions. Also, I think for each new Lambda container creation, Cloudwatch creates a new Log Stream for the Lambda function. Maybe it is possible to somehow use those.

But I was wondering if there exists a different way to get these numbers programmatically, for example like using boto3 api etc.?

Answer 1

The short answer is no.

You can programatically access cloudwatch metrics (see: boto3 CloudWatch.Client.get_metric_data ), however these metrics lag behind by a minute. Even worse, while individual lambdas return invocations , you can only get ConcurrentExecutions across the whole of your account- which means the best you could ever do is if you put a lambda in it's own AWS account , and even then you'd still be a minute behind- which is typically longer than the average lambda lifespan.

I should point out, though, that through 'reserved concurrency' lambda does expose a way to, atleast rudimentarily, control concurrency. The example use-case for this is if you are calling an external service/database that has a limited connection pool.

Answer 2

I have done this by having each running lambda instance write a status json file named {task}{request_id}.json to a specific s3 folder in a bucket that tracks lambdas. I have three sub-folders, /Running, /Completed, /Failed, so I can keep track of the total number of completions and failures. Finding out how many are running amounts to listing the files in the /Running folder, which is very fast and does not require actually fetching or opening the files.

Each lambda first creates a status file in /Running. It does its work inside a try/except block, to catch all Python exceptions, and then reads the dict in the Running folder of that request, and updates it with other information, such as total duration and any error details. It then deletes the status file in /Running and creates a status file in either /Completed or /Failed.

I will say that my application has lambda instances that last usually say 400 seconds, so the overhead of this tracking is not too bad, and it tends to run about only a few thousand lambdas in any one job. If your application launches small lambdas and at high frequency, then perhaps this is too much overhead.

In my case, requests are made explicitly in the code (rather than being triggered) but each runs asynchronously and in parallel. The AWS Lambda system will queue up requests over the concurrency limit and then throttling occurs. As soon as lambdas become available, the queued and throttled requests will launch an instance up to the concurrency limit.

I have also introduced another file called runtoken{parent_pid}.json which is established in a known s3 folder which each lambda can check to see if it is authorized to run. The {parent_pid} is provided to keep a lambda from confusing a new authorization from an older one. But basically, if for some reason I need to halt execution, all I need do is to delete the runtoken file. Each lambda within internal loops checks that the file exists and that the parent_pid (process id) matches the one it was launched with. If not, it gracefully exits and returns a successful status code to AWS lambda, while posting a status to the /Failed folder. If the lambda exits with an error, such as sys.exit(1), then the AWS lambda launcher will retry the lambda function. All lambdas check the Running folder to make sure they are not being retried. The request_id is the same in retried lambdas.