Docker + pubsub + 子进程挂起，没有错误

Question

I'm running pubsub in a docker container and am experiencing hangs after a few minutes. I've been running pubsub with similar commands outside of docker, and see no hangs (over 1 year running).

I was able to reproduce my issue with a simplified implementation outlined below.

The following four files (docker file, requirements.txt, worker.py, test.py) are placed into the same directory. They are located in this gist. Basically the worker.py is calling a subprocess function, which calls test.py to run a time.sleep(3).

Gist for code and debug outputs

To build the docker file, I placed the Dockerfile, requirements.txt, worker.py, and test.py into a directory and ran:

docker build --tag=test .

To run the docker file, I did the following:

docker run -it \
   -e GOOGLE_APPLICATION_CREDENTIALS=/keys/service-account.json \
   -e GRPC_ENABLE_FORK_SUPPORT=false \
   -e GRPC_VERBOSITY=debug \
   -e GRPC_TRACE=all \
   -v /tmp/keys/:/keys/:ro \
   -v /tmp/logs:/logs \
test:latest bash

The file service-account.json holds my pubsub keys.

I'm running docker on my Mac, but I've also experienced the same hanging using kubernetes, and on an Ubuntu 18.04 machine running docker. The issue is that after a few minutes, the pubsub subscriber becomes unresponsive. The above code runs perfectly in a virtual environment without docker.

The file (in gist) publish_output.txt is the log file of my pubsub publisher output, and containers the message ID's.

The file subscriber_output.txt (in gist) is the subscriber log output. I set GRPC_TRACE to all and GRPC_VERBOSITY to debug. The last successful message received was message ID 1253690400552289 (line 9338 of subscriber_output.txt). Messages 1253713007710668, 1253714609061509, 1253713464329969 and onward were all not received by pubsub.

Is there any additional setting I need to do? Unfortunately, I'm running custom C++ code from python so I need to use subprocesses.

The current python packages in my docker configuration:

CacheControl             0.12.6
cachetools               4.1.0
certifi                  2020.4.5.1
chardet                  3.0.4
firebase                 3.0.1
firebase-admin           4.3.0
firestore                0.0.8
google-api-core          1.18.0
google-api-python-client 1.9.1
google-auth              1.16.1
google-auth-httplib2     0.0.3
google-cloud-core        1.3.0
google-cloud-firestore   1.7.0
google-cloud-logging     1.15.0
google-cloud-pubsub      1.5.0
google-cloud-storage     1.28.1
google-resumable-media   0.5.1
googleapis-common-protos 1.52.0
grpc-google-iam-v1       0.12.3
grpcio                   1.29.0
httplib2                 0.18.1
idna                     2.9
iso8601                  0.1.12
msgpack                  1.0.0
pip                      20.1.1
protobuf                 3.12.2
pyasn1                   0.4.8
pyasn1-modules           0.2.8
pytz                     2020.1
requests                 2.23.0
rsa                      4.0
setuptools               46.4.0
six                      1.15.0
uritemplate              3.0.1
urllib3                  1.25.9
wheel                    0.34.2

Any help is appreciated.

Thanks!

Answer 1

Looks like the issue is with the google-core-api.

I was able to resolve it by following the recommendations here:

https://github.com/googleapis/python-pubsub/issues/112

Basically the requirements.txt should use these versions:

google-api-python-client <= 1.8.4
google-api-core <= 1.16.0