GAE Python Standard: Exceeded hard memory limit while RSS is twice smaller

Question

I have an app in App Engine Python3 Standard Env (Flask). The app processes images. It downloads and resizes them using temp (/tmp) space. For image processing it uses Pillow (PIL.Image) which loads everything into memory. So the app is quite memory consuming because images can be big (eg 6000x6000). I started getting "Exceeded hard memory limit" error on F1/F2/B1/B2 instances. And it was kinda understandable. So I added tracing with current value of maximum resident set size RSS using Python's resource module:

maxrss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

and increased instance class to B4 (1GB of RAM). But it didn't help. Here's log from Cloud Logging:

2021-12-09 18:17:49.352 MSK 291 - mem: 418,088
2021-12-09 18:17:49.476 MSK Exceeded hard memory limit of 1024 MB with 1075 MB after servicing 7 requests total. Consider setting a larger instance class in app.yaml.
2021-12-09 18:17:49.478 MSK [start] 2021/12/09 15:17:49.477740 Quitting on terminated signal
2021-12-09 18:17:49.483 MSK [10] [INFO] Handling signal: term
2021-12-09 18:17:49.616 MSK 292 - mem: 418,088

291 and 202 the number of iteration through all images, and mem is max_rss reported by resource module.

I can't understand why there is so big difference between the server process's RSS and the GAE service's memory as AppEngine sees it.

I understand that server code is executed inside OS and OS is inside Docker and GAE measures memory from the top, but 600MB for Linux and Docker seems too much, doesn't it?

First question: how can I understand how much memory my code actually consumes on GAE? And why is there so big difference between GAE's memory and a process' RSS?

Answer 1

Well, I found the answer what occupied memory. According to https://cloud.google.com/appengine/docs/standard/python3/using-temp-files :

All files in this directory (ie /tmp, added by me) are stored in the instance's RAM , therefore writing to /tmp takes up system memory. In addition, files in the /tmp directory are only available to the app instance that created the files.

It's so simple. So, basically the instance class' memory limit is the sum of process' memory and the all files you created in /tmp folder.