Cloud Run (Python) 中 GCS Bucket 上传调用的 Google PubSub Push SubscriptionTimeouts

Question

I'm trying to develop an image processing pipeline that takes videos uploaded to one GCS Bucket, extracts all the frames as jpg images, and uploads these images to another GCS bucket. I'm using a PubSub push subscription to trigger the cloud run service. Unfortunately, the service cannot reliably process the videos under the 10 minute max request response timeout for push subscriptions. I've tracked the issue down, and it appears that uploading the frames to GCS is causing the bottleneck. The videos contain, on average, about 28000 frames (30FPS, ~15 Minutes in length). I think this should be possible in the time provided. All services are in the same region/zone.

Is there a way to increase the throughput for these GCS blob uploads? When I use gsutil to copy a video blob from bucket to another bucket (within the same region), it takes seconds.

I've tried increasing/decreasing thread count, increasing service CPU count, and increasing service memory count. I don't see any change. GCS Rate limits for writes over 1000/Sec , but I don't think I'm anywhere near this limit yet.

My service copies the main.py script as a part of the Google Cloud Run Vision Tutorial . The only modification is to change the call to my processing routine in video.py . I've included at the bottom of the post. video.py runs the actual processing.

Cloud Run Service is provisioned with 1 CPU, 512 MiB, 15 min timeout Cloud PubSub Subscription (push subscription) 10 min timeout (maximum)

video.py:

import os
from datetime import timedelta
from concurrent.futures import ThreadPoolExecutor

import cv2

from google.cloud import storage
from google.oauth2 import service_account


def upload(blob : storage.blob.Blob, buf : "numpy.ndarray"):
    blob.upload_from_string(buf.tobytes(), content_type="image/jpeg")


def process(data : dict):
    src_client = storage.Client()
    src_bucket = src_client.get_bucket(data["bucket"])
    src_blob = src_bucket.get_blob(data["name"])

    pathname = os.path.dirname(data["name"])
    basename, ext = os.path.splitext(os.path.basename(data["name"]))

    signing_creds = \
        service_account.Credentials.from_service_account_file("key.json")

    url = src_blob.generate_signed_url(
            credentials=signing_creds,
            version="v4",
            expiration=timedelta(minutes=20),
            method="GET"
        )

    count = extract_frames(url, basename, pathname)


def extract_frames(
        signed_url : str,
        basename : str,
        pathname : str,
        dst_bucket_name : str = "extracted-frames"
    ) -> int:

    dst_client = storage.Client()
    dst_bucket = dst_client.get_bucket(dst_bucket_name)

    count = 0
    vid = cv2.VideoCapture(signed_url)

    with ThreadPoolExecutor() as executor:
        ret,frame = vid.read()

        while ret:
            enc_ret, buf = cv2.imencode(".jpg", frame)

            if not enc_ret:
                msg = f'Bad Encoding [Frame: {count:06}]'
            else:
                blob_name = f"{pathname}/{basename}-{count:06}.jpg"
                blob = dst_bucket.blob(blob_name)
                executor.map(upload, (blob, buf))

            count += 1
            ret,frame = vid.read()

    vid.release()
    return count

main.py:

import base64
import json
import os

from flask import Flask, request

# import image
import video


app = Flask(__name__)


@app.route("/", methods=["POST"])
def index():
    envelope = request.get_json()
    if not envelope:
        msg = "no Pub/Sub message received"
        print(f"error: {msg}")
        return f"Bad Request: {msg}", 400

    if not isinstance(envelope, dict) or "message" not in envelope:
        msg = "invalid Pub/Sub message format"
        print(f"error: {msg}")
        return f"Bad Request: {msg}", 400

    # Decode the Pub/Sub message.
    pubsub_message = envelope["message"]

    if isinstance(pubsub_message, dict) and "data" in pubsub_message:
        try:
            data = json.loads(base64.b64decode(pubsub_message["data"]).decode())

        except Exception as e:
            msg = (
                "Invalid Pub/Sub message: "
                "data property is not valid base64 encoded JSON"
            )
            print(f"error: {e}")
            return f"Bad Request: {msg}", 400

        # Validate the message is a Cloud Storage event.
        if not data["name"] or not data["bucket"]:
            msg = (
                "Invalid Cloud Storage notification: "
                "expected name and bucket properties"
            )
            print(f"error: {msg}")
            return f"Bad Request: {msg}", 400

        try:
            # image.blur_offensive_images(data)
            video.process(data)
            return ("", 204)

        except Exception as e:
            print(f"error: {e}")
            return ("", 500)

    return ("", 500)

Answer 1

The pattern in your case, to be scalable in case where you have longer. video in the future, is to start by splitting the video in smaller sequence (let's say 3 or 5 minutes of video) and to store these sequence in Cloud Storage.

Then a new event run a new service (or the same, according to your design) and you extract all the images. If you need to have a unique source, you can name your file to have the same prefix and thus to be able to reuse it in the subsequent processes.

---

With the same idea of parallelisation, you can also imagine to leverage the multi cpu capacity of Cloud Run to process the video chunks in parallel, in the same instance.