How to debug gunicorn and python in Google App Engine?

Question

I am running a python 3.7 application with gunicorn in Google App Engine flex. When the app starts a resource intensive task, the following gets logged in the cloud logs and I can see that the task does not get completed.

[INFO] Booting worker with pid: 46

In this thread various fixes are proposed. As mentioned in the thread, gunicorn doesn't seem to log the cause for why a worker is being booted. However, it is reported in a comment that dmesg could be used to debug the cause for the issue.

I am able to get inside the container where my app is running by following these instructions . However, when running dmesg I get an error:

dmesg: read kernel buffer failed: Operation not permitted

Any ideas on how to find the cause of gunicorn worker being booted? Might be a memory issue, but it would be good to verify the cause first.

Here are the gunicorn configs being used:

workers = 2
threads = 2
timeout = 7200
worker_class = "gthread"
loglevel = "debug"

The relevant settings in my app.yaml for Google App Engine:

env: flex

manual_scaling:
    instances: 1

resources:
    cpu: 2
    memory_gb: 12
    disk_size_gb: 20

Answer 1

Based on the OP's latest comment

I was not able to debug the problem as I had hoped, but the issue indeed turned out to be a memory issue. I used a profiler to measure the memory footprint when running the python app locally. I then changed the code to run some of the memory intensive processing in s separate process. After these changes, I was able to deploy to Google App Engine without the booting problem.

Also in similar scenarios it is possible to use GCP profiler to monitor the resources usage on App engine.