The final policy size is bigger than the limit of 20480 - AWS ELK

Question

My Use case is to stream all system logs, application logs and aws cloudtrail logs to aws elasticsearch service.

work flow is

application logs --> cloudwatch log group -->default lambda function -->aws es

now i can able to stream 40+ log groups to es. after some point of time i am trying to stream more loggroup to es that time i am unable to stream. i am getting following error

"The final policy size is bigger than the limit of 20480 "

How to increase policy size

Please help me on this.

updated:

My IAM role inline policy

{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
        ],
        "Resource": [
            "arn:aws:logs:*:*:*"
        ]
    },
    {
        "Effect": "Allow",
        "Action": "es:ESHttpPost",
        "Resource": "arn:aws:es:*:*:*"
    }
]

}

Answer 1

The role you are specifying for CloudWatch to use has too many attached policies. Review the policies and consolidate them using multiple resources for the resource attribute. Alternatively you can create a second role. I recommend the first approach.

Answer 2

You can't increase the policy size, but you can remove old ELK lambda policies and replace them with a wildcard policy. This can only be done on AWS command line, as of Aug 2019 AWS does not expose this in the web dashboard.

Three commands to do that (replace us-west-1 with your region):

List all policies:

$ aws lambda get-policy --function-name <your-ELK-lambda-name> --region us-west-1

Delete an individual policy by its statement ID - once you add wildcard below, all individual policies become redundant and can be removed:

$ aws lambda remove-permission --function-name <your-ELK-lambda-name> --statement-id <statement-id> --region us-west-1

Add the wildcard policy:

$ aws lambda add-permission --function-name <your-ELK-lambda-name> --statement-id WildcardPolicy --action "lambda:InvokeFunction" --principal "logs.us-west-1.amazonaws.com" --source-arn "arn:aws:logs:us-west-1:<your-AWS-account-number>:log-group:*" --source-account "<your-AWS-account-number>" --region us-west-1

2 more issues - as you add new logs, it will keep adding policies, so even with the wildcard policy you will have to delete new individual policies because it's not smart enough to not add them. Also, there is a UI glitch - these newly attached logs will not show up on the ELK Lambda web page properly. But at least this will help get past the policy size limit.