Randomly getting 121 error on Google App Engine

Question

I experiment same error than this question with a very small application written in Java. Randomly a servlet throws a 500 error with Error Code 121 in description but no Stack Trace.

Here is the log:

23/Jun/2013:01:37:11 -0700] "GET /premierQuestionnaire?annee=DES3&desc=false&installerLiberal=false&connaitAucun=on&roleNational=on&adhererEmblee=false HTTP/1.1" 500 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36" "these-emilien.appspot.com" ms=569 cpu_ms=0 loading_request=1 exit_code=121 app_engine_release=1.8.1 instance=00c61b117ced60a7064344269a551e9083a10fac
I 2013-06-23 10:37:11.033
   This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
W 2013-06-23 10:37:11.033
   A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. (Error code 121)

Tried to go into issue he proposed but they are restricted now,, so can't be accessed. No communication from Google. No error or system failure on status page .

If anyone has an idea, or news that would be a great thing.

Answer 1

Solution moved from @CreatixEA 's question post.

I found an answer from staff:

Beginning around 11pm on Monday, May 7th and continuing until 11am on Tuesday, May 8th, some App Engine applications saw errors marked “error code 121” in their application logs, which resulted from unnecessary instance termination.

A week prior to this issue, a change was made to the infrastructure underlying the App Engine scheduler, which disrupted our memory accounting system. This issue was slow to surface, and did not result in any serious impact to our users before our existing monitoring caught the error. To address this issue, we rolled out an alternative method for memory accounting on May 7th. This alternate method overestimated the amount of memory currently in use, and our schedulers slowly started accumulating incorrect values for memory usage.

The overestimation caused the App Engine scheduler to erroneously assume that our infrastructure was under constant memory pressure, which in turn resulted in over-aggressive termination of instances, which was visible as “error code 121” in an affected application's logs. Upon aggregating user reports of the issue on the morning of May 8th, our reliability team determined the source of the erroneous calculations, and rolled out a new fix which corrected the memory usage overestimation, and halted the unnecessary terminations.

We do not consider the time to repair an issue at this level of impact acceptable. We are adding new alerts and tools for memory accounting to prevent recurrence of similar issues in the future and to decrease our response time. Application code or Admin Console settings did not influence whether your application was affected by this issue, and no changes to your code or settings are needed.

We appreciate your patience during this issue and apologize for any inconvenience this caused you or your customers. If you feel that your paid application experienced an SLA violation, please fill out this form.

Regards, Christina Ilvento on behalf of the Google App Engine Team