如何将 tensorflow 保存的 model 加载到 memory 中，并且在谷歌应用引擎中部署后无需再次加载？

Question

I am using a tensorflow hub model "Universal Sentence Encoder" and sometimes app engine shutdowns automatically and it takes more time to load this model again. How to make that model stay in memory?

   runtime: python
    env: flex
    
    runtime_config:
        python_version: 3
    
    automatic_scaling:
      min_num_instances: 1
      max_num_instances: 1
      cpu_utilization:
        target_utilization: 1
    readiness_check:
      app_start_timeout_sec: 1800
    liveness_check:
      path: "/liveness_check"
      check_interval_sec: 30
    resources:
        cpu: 1
        memory_gb: 6
        disk_size_gb: 15                        
    entrypoint: gunicorn -k uvicorn.workers.UvicornWorker -w 4 app.main:app --timeout 1000

Logs:

A 2021-03-22T10:53:59Z [2021-03-22 10:53:59 +0000] [8] [INFO] Started server process [8]
 
A 2021-03-22T10:53:59Z [2021-03-22 10:53:59 +0000] [8] [INFO] Waiting for application startup.
 
A 2021-03-22T10:53:59Z [2021-03-22 10:53:59 +0000] [8] [INFO] Application startup complete.
 
A 2021-03-22T11:01:03Z [2021-03-22 11:01:03 +0000] [1] [INFO] Handling signal: term
 
A 2021-03-22T11:01:03Z [2021-03-22 11:01:03 +0000] [8] [INFO] Shutting down
 
A 2021-03-22T11:01:03Z [2021-03-22 11:01:03 +0000] [8] [INFO] Error while closing socket [Errno 9] Bad file descriptor
 
A 2021-03-22T11:01:04Z [2021-03-22 11:01:04 +0000] [8] [INFO] Waiting for application shutdown.
 
A 2021-03-22T11:01:04Z [2021-03-22 11:01:04 +0000] [8] [INFO] Application shutdown complete.
 
A 2021-03-22T11:01:04Z [2021-03-22 11:01:04 +0000] [8] [INFO] Finished server process [8]
 
A 2021-03-22T11:01:04Z [2021-03-22 11:01:04 +0000] [8] [INFO] Worker exiting (pid: 8)
 

Answer 1

With App Engine flex your application is started when deployed (because the min instance is 1, you always have 1 instance started). In your startup routine, add the model loading in memory and keep it in global variable. Like this, any requests that you receive on your service will be able to use it.

However some caveats:

• App Engine flex is restarted at least once a week to update the underlying platform. So, once a week, at least, your instance will restart. But because you load your modal at startup, no worries about the request response time!
• When the service scale up (not your case), a new instance is created and thus the model is again loaded in memory
• Similarly, if your instance crash (out of memory for example, or unhandled exception), a new one is created and thus the model is loaded again at this time.