Memory error deploying app in Google App Engine

Question

I'm trying to deploy a flask API in Google App Engine but faced MemoryError.

When running gcloud app deploy I receive the following message via terminal:

Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build f857cba3-1bf5-4384-acec-f1ee3ba97c8b status: FAILURE.
Build error details: {"error":{"errorType":"BuildError","canonicalCode":"INVALID_ARGUMENT","errorId":"26CADD66","errorMessage":"14 May 2020 13:48:07 INFO     Arguments: ['--parser_script=/usr/local/bin/ftl.par', '--src=', '--entrypoint=', '--name=us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0', '--directory=/workspace', '--destination=/srv', '--cache-repository=us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00']\n14 May 2020 13:48:07 INFO     Unparsed arguments: ['--name=us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0', '--directory=/workspace', '--destination=/srv', '--cache-repository=us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00']\n14 May 2020 13:48:07 INFO     Executing ['/usr/local/bin/ftl.par', '--name=us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0', '--directory=/workspace', '--destination=/srv', '--cache-repository=us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00']\nINFO     FTL version python-v0.17.0\nINFO     Beginning FTL build for python\nINFO     FTL arg passed: virtualenv_dir /env\nINFO     FTL arg passed: ttl 168\nINFO     FTL arg passed: python_cmd /opt/python3.7/bin/python3.7\nINFO     FTL arg passed: cache True\nINFO     FTL arg passed: virtualenv_cmd virtualenv\nINFO     FTL arg passed: entrypoint None\nINFO     FTL arg passed: exposed_ports None\nINFO     FTL arg passed: pip_cmd /env/bin/python3.7 -m pip\nINFO     FTL arg passed: tar_base_image_path None\nINFO     FTL arg passed: export_cache_stats False\nINFO     FTL arg passed: builder_output_path \"\"\nINFO     FTL arg passed: destination_path /srv\nINFO     FTL arg passed: sh_c_prefix False\nINFO     FTL arg passed: base us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00\nINFO     FTL arg passed: cache_key_version v0.17.0\nINFO     FTL arg passed: cache_salt \nINFO     FTL arg passed: cache_repository us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d\nINFO     FTL arg passed: venv_cmd /opt/python3.7/bin/python3.7 -m venv /env\nINFO     FTL arg passed: name us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0\nINFO     FTL arg passed: global_cache False\nINFO     FTL arg passed: upload True\nINFO     FTL arg passed: fail_on_error True\nINFO     FTL arg passed: output_path None\nINFO     FTL arg passed: directory /workspace\nINFO     FTL arg passed: additional_directory /.googleconfig\nINFO     FTL arg passed: verbosity DEBUG\nINFO     starting: full build\nINFO     starting: builder initialization\nINFO     Loading Docker credentials for repository 'us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00'\nINFO     Loading Docker credentials for repository 'us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0'\nINFO     builder initialization took 0 seconds\nINFO     starting: build process for FTL image\nINFO     starting: checking_cached_interpreter_layer\nINFO     starting: check python version\nINFO     `python version` full cmd:\n/opt/python3.7/bin/python3.7 --version\nINFO     `python version` stderr:\n\nINFO     check python version took 0 seconds\nDEBUG   .

When checking the build details on GCP I got this:

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/ftl.par/__main__.py", line 65, in <module>
  File "/usr/local/bin/ftl.par/__main__.py", line 54, in main
  File "/usr/local/bin/ftl.par/__main__/ftl/python/builder.py", line 123, in Build
  File "/usr/local/bin/ftl.par/__main__/ftl/common/layer_builder.py", line 42, in BuildLayer
  File "/usr/local/bin/ftl.par/__main__/ftl/common/ftl_util.py", line 116, in zip_dir_to_layer_sha
MemoryError

My requirements.txt is as follows:

scikit-learn==0.20.3
pandas==1.0.3
xgboost==1.0.2
fuzzywuzzy==0.18.0
python-Levenshtein==0.12.0
gensim==3.8.1
nltk==3.3
Flask==1.1.2
flask-cors==3.0.7
requests==2.23.0

My app.yaml is as follows:

runtime: python37
instance_class: F4_1G
resources:
  cpu: 3
  memory_gb: 16
  disk_size_gb: 30

I tried a lot of configurations related to cpus and memory and none worked.

I think the problem is I'm trying to load a word2vec model with 3.5gb:

model = gensim.models.KeyedVectors.load_word2vec_format('./GoogleNews-vectors-negative300.bin', binary=True)

Answer 1

In App Engine the app.yaml can use one of the tags between instance_class and resources as both define the machine type that will be assigned to the app engine instance.

In App Engine Standar the tag used is ´instance_class´ as mentioned here all the isnstances which have an instance class come with 1 GB of disk then it makes scence to be getting an error when trying to upload a file bigger than this 1 GB of storage.

If you want to specify more about the machine type that is going to be used like the memory, disk size etc then you need to deploy in [App Engine Flex] where you can specify the resources tag.

To deploy on flex the app.yaml needs to specify that the environment is flex as well as the custom configuration for the instance in this case.

runtime: python
env: flex

runtime_config:
  python_version: 3

resources:
  cpu: 4
  memory_gb: 16
  disk_size_gb: 30

I modified the CPUs from 3 to 4 as this value shall be 1 or an even number.

Consider that App Engine Flex is more expensive than standard and takes more time to deploy.

Sometimes this can send a Error Response: [4] Timed out waiting for the app infrastructure to become healthy. error, it is expected that it can be thrown, if you get this issue try again some time after and if it is happening again please try deploying the app in a different region.