Mitigating storage issues with catalina.out on Elastic Beanstalk

Question

I'm having an issue where the catalina.out log file on a Tomcat 8 Java 8 Elastic Beanstalk instance is expanding to the point where there's no storage left in the instance, which does not allow the instance to keep sending logs to CloudWatch, and prevents me from deploying new versions. The catalina.out file is forwarded to CloudWatch via an .ebextension.

Is there any way to mitigate this? some theoretical ways might be via rolling logs and deleting the old versions of catalina.out , or rolling the instances when they have no storage space left.

Before answering, please consider the following:

• I'm aware that the log should not be as large as it is, this currently is a limitation and cannot be fixed in the near future or before fixing this issue.

• I'm already using scaling based on a different parameter, so scaling based on storage space is not available to me.

• I'm using Log4j2, but catalina.out is handled by tomcat so configuring a rolling file with deletion is not an option (via Log4j2 specifically, there may be a way to do it via Tomcat on AWS).

• I'd like to keep the catalina.out file on the instances in addition to forwarding it's contents to CloudWatch rather than directly sending logs to CloudWatch.

• There's no way to know the maximum storage the log file will take as you don't always know in advance when a new instance will be used (so expanding storage isn't a long term solution).

Answer 1

• Separate the log files in a different partition, ephemeral/instance storage can be good for this. So even if it fills, do not damage other operations: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html

• Ship and rotate logs with CloudWatch Logs Agent: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html

• Set environment, JVM, Log4J and container settings using .ebextensions: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html