Figuring out the real Java heap on AWS Fargate

Question

We have a container (task) running in AWS Fargate and we see some weird behavior where memory don't seem to be utilized correctly so I've been trying to figure out how this is actually set/used...

The task is set to 2vCPU/4GB memory.

The app is a Tomcat running on Java and memory is set to 80% for Java ( -XX:MaxRAMPercentage=80.0 ) but somehow this is seems to be altered by Tomcat to:

echo $CATALINA_OPTS
-Xms1g -Xmx4g -Djava.security.egd=file:/dev/./urandom

Tomcat/Catalina thus shows Xmx4g for the container with max. 4GB memory.

However trying to figure out what is really used in the container has proven a bit difficult but looking in the container itself:

root@ip-172-31-28-76:/home/git# java -XX:+PrintFlagsFinal -version | grep HeapSize
   size_t ErgoHeapSizeLimit                        = 0                                         {product} {default}
   size_t HeapSizePerGCThread                      = 43620760                                  {product} {default}
   size_t InitialHeapSize                          = 67108864                                  {product} {ergonomic}
   size_t LargePageHeapSizeThreshold               = 134217728                                 {product} {default}
   size_t MaxHeapSize                              = 1073741824                                {product} {ergonomic}
    uintx NonNMethodCodeHeapSize                   = 5825164                                {pd product} {ergonomic}
    uintx NonProfiledCodeHeapSize                  = 122916538                              {pd product} {ergonomic}
    uintx ProfiledCodeHeapSize                     = 122916538                              {pd product} {ergonomic}
openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment 18.9 (build 11.0.13+8)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing)

Here it looks like the MaxHeapSize is 1GB (1073741824 bytes)

Outputting the docker stats from the container through Fargate gives me a Limits.Memory: 4096 as expected but then the memory stats are werird:

"memory_stats": {
        "usage": 541478912,
        "max_usage": 546689024,
        "stats": {
            "active_anon": 531214336,
            "active_file": 249856,
            "cache": 270336,
            "dirty": 0,
            "hierarchical_memory_limit": 4294967296,
            "hierarchical_memsw_limit": 9223372036854771712,
            "inactive_anon": 0,
            "inactive_file": 3846144,
            "mapped_file": 0,
            "pgfault": 156816,
            "pgmajfault": 0,
            "pgpgin": 149259,
            "pgpgout": 18544,
            "rss": 535040000,
            "rss_huge": 0,
            "total_active_anon": 531214336,
            "total_active_file": 249856,
            "total_cache": 270336,
            "total_dirty": 0,
            "total_inactive_anon": 0,
            "total_inactive_file": 3846144,
            "total_mapped_file": 0,
            "total_pgfault": 156816,
            "total_pgmajfault": 0,
            "total_pgpgin": 149259,
            "total_pgpgout": 18544,
            "total_rss": 535040000,
            "total_rss_huge": 0,
            "total_unevictable": 0,
            "total_writeback": 0,
            "unevictable": 0,
            "writeback": 0
        },
        "limit": 9223372036854771712
    }

What gives here?

Is the "true" max heap only 1GB as shown in grep HeapSize ?

EDIT #1 : Running the container command for stats for Fargate curl ${ECS_CONTAINER_METADATA_URI_V4}/stats I can see that "max. memory used" is actually 2.4GB so the shown MaxHeapSize as 1GB is not correct!

EDIT #2 : Mangaed to get it to accept -Xms1g -XX:MaxRAMPercentage=80.0 after some testing and now MaxHeapSize shows as 2GB. I've up'ed the container to 2vCPU/8GB but the running mem stats still shows that max mem used is 2.4GB so regardless of settings/changes it is not using more than the 2.4GB...

Answer 1

Suggest to investigate the meaning of metaspace .

Configure metaspace with JVM option: -XX:MetaspaceSize

Suggest to follow this formula:

Available RAM = 512M(for OS) + 256M(for Tomcat metaspace) + JVM heap