How to analyze actual memory usage on App Engine Java?

Question

I have an App Engine Java application that runs on automatically scaled F1 instances (600 Mhz, 128 MB RAM). After warm-up, the memory usage per instance as shown by the dashboard stays at about 220 MB. According to the docs , this includes "runtime overhead".

• Am I right to assume that this number reflects the maximum permgen and heap size, but not necessarily the actually used one? I have never received an out-of-memory error.
• What is the best way to analyze the memory usage in the production environment? Is there a way to perform a heap dump despite App Engine's runtime restrictions?

Local analysis using, eg, VisualVM does not provide an accurate picture since it includes the development server, in-memory versions of the Datastore and Memcache, etc. In total, it adds up to 8,000 loaded classes (including the dev server, my own classes, and about 85 libraries), a resulting permgen size of 50 MB, and a minimum heap size of 50 MB after a full garbage collection.

Some background on my question: The scheduler sometimes (often during the first few requests following the completion of a regular synchronization job) starts up a new instance to process a request load that is usually handled by one instance only. After a few more requests, all requests will be routed entirely and consistently to the first instance again. Based on the set of scheduler rules outlined here , I am trying to figure out whether this behavior might be due to the scheduler predicting a (temporary) lack of instance memory to process additional requests.

Answer 1

What is the best way to analyze the memory usage in the production environment? Is there a way to perform a heap dump despite App Engine's runtime restrictions?

There's no good way. Instrumentation is not well supported.