AWS Cloudfront returns 403 when PUT request body is greater than 8kb

Question

I have an API behind AWS Cloudfront which has functioned fine with our front end application for years. Recently, after a feature release, I've noticed some users are reporting data not being saved and the app just hanging. After a lengthy investigation, I've discovered that our Cloudfront distribution will return a 403 Forbidden error when a PUT request's JSON body is greater than 8kb. Anything less works fine, anything more returns 403. I verified this by sending PUT requests with a decreasing body data size until I got the expected 201 Created response, and just checked the size of the body sent. The JSON is properly formatted.

The error returns in about 170ms and contains the header from Cloudfront X-Cache: Error from cloudfront

I have looked for settings on size limits, I've tried disabling the WAF rules, I've tried "Compress objects" to both on/off. Would having Real-time logs enabled have an impact on the max data accepted somehow? Seems crazy but I'm kind of bewildered by this issue.

I would love to show you all some kind of log from Cloudwatch to help, but requests that don't make it past Cloudfront are not logged in Cloudwatch, and I have not been able to setup any kind of logging to get better insight into why it's barfing.

Answer 1

As was the initial hunch, this turned out to be a WAF ACL rule issue.

The blocking ACL was applied to the application load balancer, so finding it in the Web ACL list either requires inspecting the region where your load balancer is (eg us-west-2), or by inspecting the load balancer's Integrate Services, where you can see any AWS WAF rules:

• AWS > EC2 > Load Balancers > {instance} > Integrated services (tab) > AWS WAF

The specific rule was in an AWS managed rule set called AWS-AWSManagedRulesCommonRuleSet . Just turn the SizeRestrictions_BODY rule to Count instead of "Use action defined in the rule"

This obviously has impacts on what requests get through to your application, so do with that what you will.