Mi GCP 数据流步骤未启动

Question

我有这个简单的 GCP 数据流，它从 pubsub 读取，然后打印一条日志，指示消息到达。

import logging
import os
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions, StandardOptions

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()
logger.setLevel(logging.INFO)

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "./credentials.json"
INPUT_SUBSCRIPTION = "projects/PROJECT_ID/subscriptions/SUBSCRIPTION_NAME"


class FromProtoToJson(beam.DoFn):

    def to_runner_api_parameter(self, unused_context):
        return "beam:transforms:custom_parsing:custom_v0", None

    def process(self, element: bytes, timestamp=beam.DoFn.TimestampParam, window=beam.DoFn.WindowParam):
        print("element arrived")


def run():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--input_subscription",
        help='Input PubSub subscription of the form "projects/<PROJECT>/subscriptions/<SUBSCRIPTION>."',
        default=INPUT_SUBSCRIPTION
    )

    known_args, pipeline_args = parser.parse_known_args()

    pipeline_options = PipelineOptions(pipeline_args)
    pipeline_options.view_as(StandardOptions).streaming = True

    with beam.Pipeline(options=pipeline_options) as p:
        (
            p
            | "ReadFromPubSub" >> beam.io.gcp.pubsub.ReadFromPubSub(
                subscription=known_args.input_subscription, timestamp_attribute=None
            )
            | "FromProtoToJson" >> beam.ParDo(FromProtoToJson())
        )

if __name__ == "__main__":
    run()

如您所见，它有 2 个步骤，但它们没有启动，此数据流由使用 apache 光束 2.41.0 的命令启动：

python realtime_pipeline.py --streaming --input_subscription projects/my-project/subscriptions/test-sub --runner DataflowRunner --project my-project --region us-west4 --temp_location gs://my-project-pipeline-bucket --job_name test --max_num_workers 1

当我在获得所有日志后执行该命令时：工作人员已成功启动

部署和数据流仪表板中均未显示任何错误，但是当我将 go 转到日志资源管理器->日志仪表板时，我在 GCE 错误中看到此错误：

{
    insertId: "1qzq6lgfm8cwty"
    jsonPayload: {
        localTimestamp: "2022-09-27T10:42:04.7643Z"
        message: "Error updating SSH keys for root: mkdir /root/.ssh: read-only file system."
    }
    logName: "projects/my-project/logs/GCEGuestAgent"
    receiveTimestamp: "2022-09-27T10:42:05.773260545Z"
    resource: {2}
    severity: "ERROR"
    sourceLocation: {3}
    timestamp: "2022-09-27T10:42:04.764323945Z"
}

我怎样才能让它工作？，在消息“工人已经成功开始”之后，工作似乎已经准备好了，但没有被 pubsub 触发......谢谢你的帮助。