Google App Engine “Handling signal: term” everytime

Question

I'm facing a problem with google app engine. Like on this post : 63392824

My flask application is triggered by POST request, do some stuffs during maybe one hour and that's all.

After some times I see that my new requests are stopped without error. This is what I see on the logs :

[..some logs without errors..] 
    2020-09-07 07:24:09 app-engine-name[20200903t092926]  [2020-09-07 07:24:08 +0000] [10] [INFO] Handling signal: term
[...]
    2020-09-07 07:24:10 app-engine-name[20200903t092926]  [2020-09-07 07:24:10 +0000] [21] [INFO] Worker exiting (pid: 21)
    2020-09-07 07:24:10 app-engine-name[20200903t092926]  [2020-09-07 07:24:10 +0000] [25] [INFO] Worker exiting (pid: 25)

How can I debbug this and find the reason of this term signal ?

Thanks in advance for your time !

EDIT

I try to upscale my app to F4_1G (2G memory limit) but the problem is steel appearing. The memory seems to not be the problem like I see on Googe monitoring :

Answer 1

This signal is sent by App Engine agent, usually when your app is idle to recreate an instance. This is a common routine meant to allocate shared resources in a most efficient way.

The fact that you are seeing it too often most likely means that your app is leaking memory and the agent kills the underlying instance when it reaches the limit and recreates it. This is common issue for Java apps.

Allocated memory depends on instance class in Standard. And in Flex you allocate resources through app.yaml . You can observe resource usage either in App Engine Dashboard or in Cloud Monitoring -> Metrics Explorer.

Answer 2

After searching for multiple weirds solutions I found some ways to make it works.

First thing : there is more logs directly on the Google App Engine web interface than in the gutils command. The lack of memory error message can be found here.

Second thing : instances is automaticly stop after some non-calling time. Bascily if not POST / GET requests is made within the idle_timeout the instance will be stopped. But if, likes me, your instance is working on long-time task the instance will be stopped and your task will be unfairly stopped.

So the idea is to make false GET request to keep your instance alive.

Something like that (Maybe you'll have to adapt the url) :

import os, requests

requests.get( 'https://{}-dot-{}-dot-{}.ew.r.appspot.com/keep-alive'.format( os.environ['GAE_VERSION'],os.environ['GAE_SERVICE'], os.environ['GOOGLE_CLOUD_PROJECT'] ) )

The API function don't have to be complex :

@api_blueprint.route('/keep-alive', methods=['GET'])
def keep_me_alive():
    resp = make_response("I'm alive ! o/") 
    resp.headers['Access-Control-Allow-Origin'] = '*'
    return resp

With the integration of the GAE_VERSION param, you'll be sure to keep-alive the right one instance.

Call this request line regularly during your task and stop doing it at this end to let your instance be stopped after.

EDIT: This is working if you have only one instance by version . And it's impossible to call with the instance id in standard environemnt. :(