Pub/Sub on Google App Engine runs at ~100% CPU

Question

I've been trying to test a project that I was going to set up on the App Engine. I can't specify my exact code because of confidentiality, but it's customized between the tutorials: Scalable Video Transcoding ( Github ) and Using Cloud Pub/Sub with Python ( Github )

Essentially it runs an App Engine service with Flask to handle incoming requests which enqueue a task through psq, and a worker service that runs psqworker to execute the tasks.

Testing the services themselves worked perfectly. The media I was using was transcoded in the worker service and returned back to my cloud storage.

The problem is that after an hour from starting, whether I queue any tasks or not, each worker instance starts to ramp up to 99-100% CPU usage. When I SSH into an instance, psqworker is the cause. This is really bad on App Engine because it wants to scale and add more instances (which in turn do the same after an hour of starting). I attempted to go through the Stackdriver logs but couldn't find any obvious cause.

I'm also not really sure why Ruby is running on the CPU as well, seen in the screenshot. It's meant to be a custom flexible runtime based off Google's python image.

I ran the services locally on my Windows machine and there was no spike in CPU usage.

The Dockerfile:

# The Google App Engine python runtime is Debian Jessie with Python installed
# and various os-level packages to allow installation of popular Python
# libraries. The source is on github at:
#   https://github.com/GoogleCloudPlatform/python-docker
FROM gcr.io/google_appengine/python

RUN apt-get -y update && apt-get install -y libav-tools

# Create a virtualenv for dependencies. This isolates these packages from
# system-level packages.
RUN virtualenv /env -p python3.6

# Setting these environment variables are the same as running
# source /env/bin/activate.
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH

# Copy the application's requirements.txt and run pip to install all
# dependencies into the virtualenv.
ADD requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt

# # Add the application source code.
ADD . /app

CMD mkdir /tmp/audio

# CMD honcho start -f /app/procfile transcoder
CMD honcho start -f /app/procfile worker monitor

Screenshot of CPU usage after I noticed what was going on:

Answer 1

After reaching out to one of the developers, it turns out the CPU usage is due to some issues with the grpc library on Linux and how their source is distributed.

Workaround found in Github issue

When installing required packages, this can be remedied by installing over the grpcio package through pip by building it instead of just getting the binary.

In my Dockerfile I added the line after installing the rest of the requirements as in the tutorial:

RUN pip install grpcio --ignore-installed --no-binary grpcio

The CPU issue seemed to be resolved.