如何在 Google Colab 上创建火炬服务器并使用预测

Question

I try to create a torchserve on google colab but it took forever to load and it seem that i can't even connect to the serve. Is this possible to create a torchserve on colab? Here is what it show when i try to do it.

Removing orphan pid file. WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. 2022-05-01T10:56:00,634 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager... 2022-05-01T10:56:00,775 [INFO ] main org.pytorch.serve.ModelServer - Torchserve version: 0.5.3 TS Home: /usr/local/lib/python3.8/site-packages Current directory: /content Temp directory: /tmp Number of GPUs: 0 Number of CPUs: 2 Max heap size: 3248 M Python executable: /usr/local/bin/python Config file: /content/drive/MyDrive/COferenceGerman/config.properties Inference address: http://0.0.0.0:8000 Management address: http://0.0.0.0:8001 Metrics address: http://0.0.0.0:8002 Model Store: /content/drive/MyDrive/COferenceGerman Initial Models: droc_incremental_no_segment_distance.mar Log dir: /content/logs Metrics dir: /content/logs Netty threads: 0 Netty client threads: 0 De fault workers per model: 2 Blacklist Regex: N/A Maximum Response Size: 6553500 Maximum Request Size: 6553500 Limit Maximum Image Pixels: true Prefer direct buffer: false Allowed Urls: [file://. |http(s)?://. ] Custom python dependency for model allowed: false Metrics report format: prometheus Enable metrics API: true Workflow Store: /content/drive/MyDrive/COferenceGerman Model config: N/A 2022-05-01T10:56:00,784 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin... 2022-05-01T10:56:00,824 [INFO ] main org.pytorch.serve.ModelServer - Loading initial models: droc_incremental_no_segment_distance.mar 2022-05-01T10:56:13,006 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 0.0.1 for model droc_incremental_no_segment_distance 2022-05-01T10:56:13,009 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 0.0.1 for Z20F35E630DAF44DBFA4C3F68F5399D8C Z droc_incremental_no_segment_distance 2022-05-01T10:56:13,010 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model droc_incremental_no_segment_distance loaded. 2022-05-01T10:56:13,011 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: droc_incremental_no_segment_distance, count: 2 2022-05-01T10:56:13,032 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2022-05-01T10:56:13,039 [DEBUG] W-9000-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [python3, /usr/local/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000] 2022-05-01T10:56:13,051 [DEBUG] W-9001-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [python3, /usr/local/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9001] 2022-05-01T10:56:13,318 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - Traceback (most recent call last): 2022-05-01T10:56:13,327 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - File "/usr/local/lib/python3.8/site-packages/ts/model_service_worker.py", line 17, in 2022-05-01T10:56:13,329 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - from ts.model_loader import ModelLoaderFactory 2022-05-01T10:56:13,331 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - File "/usr/local/lib/python3.8/site-packages/ts/model_loader.py", line 16, in 2022-05-01T10:56:13,333 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - from ts.service import Service 2022-05-01T10:56:13,333 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - File "/usr/local/lib/python3.8/site-packages/ts/service.py", line 14, in 2022-05-01T10:56:13,339 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - from ts.protocol.otf_message_handler import create_predict_response 2022-05-01T10:56:13,340 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - File "/usr/local/lib/python3.8/site-packages/ts/protocol/otf_message_handler.py", line 15, in 2022-05-01T10:56:13,341 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - import torch 2022-05-01T10:56:13,347 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://0.0.0.0:8000 2022-05-01T10:56:13,349 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel. 2022-05-01T10:56:13,350 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch' 2022-05-01T10:56:13,367 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://0.0.0.0:8001 2022-05-01T10:56:13,367 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel. 2022-05-01T10:56:13,369 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://0.0.0.0:8002 2022-05-01T10:56:13,373 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - Traceback (most recent call last): 2022-05-01T10:56:13,377 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - File "/usr/local/lib/python3.8/site-packages/ts/model_service_worker.py", line 17, in 2022-05-01T10:56:13,377 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - from ts.model_loader import ModelLoaderFactory 2022-05-01T10:56:13,378 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - File "/usr/local/lib/python3.8/site-packages/ts/model_loader.py", line 16, in 2022-05-01T10:56:13,378 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - from ts.service import Service 2022-05-01T10:56:13, 378 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - File "/usr/local/lib/python3.8/site-packages/ts/service.py", line 14, in 2022-05-01T10:56:13,378 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - from ts.protocol.otf_message_handler import create_predict_response 2022-05-01T10:56:13,379 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - File "/usr/local/lib/python3.8/site-packages/ts/protocol/otf_message_handler.py", line 15, in 2022-05-01T10:56:13,379 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - import torch 2022-05-01T10:56:13,379 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch' 2022-05-01T10:56:13,400 [INFO ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9001-droc_incremental_no_segment_distance_0.0. 1-stderr 2022-05-01T10:56:13,403 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9001-droc_incremental_no_segment_distance_0.0.1-stderr 2022-05-01T10:56:13,400 [INFO ] W-9001-droc_incremental_no_segment_distance_0.0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9001-droc_incremental_no_segment_distance_0.0.1-stdout 2022-05-01T10:56:13,404 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9001-droc_incremental_no_segment_distance_0.0.1-stdout 2022-05-01T10:56:13,428 [INFO ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-droc_incremental_no_segment_distance_0.0.1-stderr 2022-05-01T10:56:13,428 [INFO ] W-9000-droc_increment al_no_segment_distance_0.0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-droc_incremental_no_segment_distance_0.0.1-stdout 2022-05-01T10:56:13,430 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-droc_incremental_no_segment_distance_0.0.1-stderr 2022-05-01T10:56:13,430 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-droc_incremental_no_segment_distance_0.0.1-stdout 2022-05-01T10:56:13,431 [ERROR] W-9000-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerThread - Backend worker error org.pytorch.serve.wlm.WorkerInitializationException: Backend stream closed. at org.pytorch.serve.wlm.WorkerLifeCycle.startWorker(WorkerLifeCycle.java:139) ~[model-server.jar:?] at org.pytorch.serve.wlm.WorkerThread.connect(WorkerThread.java:292) ~[model-server.jar:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:179) [model-server.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util .concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2022-05-01T10:56:13,408 [ERROR] W-9001-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerThread - Backend worker error org.pytorch.serve.wlm.WorkerInitializationException: Backend stream closed. at org.pytorch.serve.wlm.WorkerLifeCycle.startWorker(WorkerLifeCycle.java:139) ~[model-server.jar:?] at org.pytorch.serve.wlm.WorkerThread.connect(WorkerThread.java:292) ~[model-server.jar:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:179) [model-server.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util .concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] Model server started. 2022-05-01T10:56:13,759 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet. 2022-05-01T10:56:13,833 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:100.0|#Level:Host|#hostname:649a8b344986,timestamp:1651402573 2022-05-01T10:56:13,835 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:66.32177734375|#Level:Host|#hostname:649a8b344986,timestamp:1651402573 2022-05-01T10:56:13,837 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:41.377681732177734|#Level:Host|#hostname:649a8b344986,timestamp:1651402573 2022-05-01T10:56:13,838 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:38.4|#Level:Host|#hostname:649a8b344986,timestamp:1651402573 2022-05-01T10:56:13,839 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:11904.3984375|#Level:Host|#hostname:649a8b344986,timestamp:1651402573 2022-05-01T10:56:13,840 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:794.34375|#Level:Host|#hostname:649a8b344986,timestamp:1651402573 2022-05-01T10:56:13,841 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:8.3| #Level:Host|#hostname:649a8b344986,timestamp:1651402573

And when i try to predict It just load forever

Any idea how can i fix this on colab? It work normal on my pc but i need it on colab.

Answer 1

It says that there is an error

ModuleNotFoundError: No module named 'torch' 

Even if you fix it. It will be tricky to run a server on Google Colab as google prohibits such usage of colab resources and they disabled ssh and other ways you can use torchserve as a server. However, if the client process is on colab as well I think if you fix the errors it might work.