App Engine error after Python web app is deployed - BlockingIOError: [Errno 11] Resource temporarily unavailable

Question

I am running my web app on a standard environment on Python 3.7 and when I test it locally everything works fine. However, after I deploy my app I am receiving the following error when the app tries to save files to the /tmp location that I have designated:

BlockingIOError: [Errno 11] Resource temporarily unavailable

This is how am designating the path to save as well as the file name.

file_num = os.urandom(10).hex()

handle, path = tempfile.mkstemp()

ytdl_format_options = {'format': 'bestaudio/best','outtmpl':path + 'song'+ file_num +'.mp3', 'quiet': True}

I'm not sure what is causing this error or if there is some type of config file setting that I am missing either on app.yaml or on gunicorn. I also tried locating the gunicorn files on the glcoud shell but they were not showing up in the directory.

I have noticed that my app will randomly work correctly when I commit new changes to files and re-deploy the app, however it will only work once and then if I immediately try running the function again it fails. This is the error I am receiving:

Traceback (most recent call last):
  File "/env/lib/python3.7/site-packages/gunicorn/workers/gthread.py", line 279, in handle
    keepalive = self.handle_request(req, conn)
  File "/env/lib/python3.7/site-packages/gunicorn/workers/gthread.py", line 348, in handle_request
    six.reraise(*sys.exc_info())
  File "/env/lib/python3.7/site-packages/gunicorn/six.py", line 625, in reraise
    raise value
  File "/env/lib/python3.7/site-packages/gunicorn/workers/gthread.py", line 331, in handle_request
    resp.write_file(respiter)
  File "/env/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 403, in write_file
    if not self.sendfile(respiter):
  File "/env/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 393, in sendfile
    sent += sendfile(sockno, fileno, offset + sent, count)
BlockingIOError: [Errno 11] Resource temporarily unavailable

Answer 1

From Filesystem :

The runtime includes a full filesystem. The filesystem is read-only except for the location /tmp , which is a virtual disk storing data in your App Engine instance's RAM.

So:

• using any location other than /tmp won't work
• the amount of available space is limited. Because of this your app needs to take care of deleting the temporary files after use. The behaviour you describe suggests the available space may only be enough for 1 of your files.

Another thing to consider is multiple parallel requests: if each of such request could create its own file, then the error could be hit because multiple files would be written simultaneously by different such requests. Adding a scheme limiting the number of requests processed in parallel would be a possible approach.

While not 100% certain I suspect the limit may also be dependent on the instance's total amount of RAM, in which case your app's configured instance class would matter as well. To check if this is true just temporarily configure an instance class with more RAM and see if that helps (without cleaning up the files maybe it'd work more than once before hitting the error, for example).

If the above suspicion is confirmed then choosing an appropriate instance class could also be an approach for addressing the problem.

Despite applying all the above it may still be possible to hit the issue, for example if some of the files you're writing are simply too big. If so then using the local filesystem is not appropriate for your app. A possible alternative would be to store your files in Google Cloud Storage (GCS) instead.