当我 stream 多个 cloudwatch 日志组到一个 lambda 时，如何增加策略限制？

Question

I have setup a lambda function to be triggered by many cloudwatch log groups. In order to do that, I added the invoke function permission on log group aws lambda add-permission and add subscription as lambda as destination aws logs put-subscription-filter . There are hundreds of log groups I need to stream to one lambda which makes the lambda trigger policy very big.

There two commands in this flow aws lambda add-permission and aws logs put-subscription-filter . I need to run these two commands per each log group. I added 46 cloudwath log groups as trigger for the lambda but when adding the 47th I got an error.

The error I got was this command:

aws lambda add-permission --function-name $AGGREGATOR_NAME \
    --statement-id add-permission-$lambdaName --action lambda:InvokeFunction \
    --principal logs.ap-southeast-2.amazonaws.com \
    --source-arn $logArn

An error occurred (PolicyLengthExceededException) when calling the AddPermission operation: The final policy size (20623) is bigger than the limit (20480).
arn:aws:logs:ap-southeast-2:***

Is there a way to get around of that? Is this a right way to stream hundreds of log groups to one lambda?

I have tried to use wildcard in the command but got a validation error.

aws lambda add-permission --function-name $AGGREGATOR_NAME --statement-id $ID --action lambda:InvokeFunction --principal logs.ap-southeast-2.amazonaws.com --source-arn "arn:aws:logs:*:*:log-group:/aws/lambda/hello*:*"

An error occurred (ValidationException) when calling the AddPermission operation: 1 validation error detected: Value 'arn:aws:logs:*:*:log-group:/aws/lambda/hello*:*' at 'sourceArn' failed to satisfy constraint: Member must satisfy regular expression pattern: arn:(aws[a-zA-Z0-9-]*):([a-zA-Z0-9\-])+:([a-z]{2}((-gov)|(-iso(b?)))?-[a-z]+-\d{1})?:(\d{12})?:(.*)

Answer 1

A way to get around of this is to utilize wildcard in --source-arn . By doing this, you don't need one lambda resource-based policy for each CloudWatch log group. Of course, the simplest way is to allow all log groups to execute lambda:InvokeFunction . In this case, you just need

aws lambda add-permission --function-name $AGGREGATOR_NAME \
    --statement-id add-permission-$lambdaName --action lambda:InvokeFunction \
    --principal logs.ap-southeast-2.amazonaws.com

Notice that the --source-arn is removed.