由於驗證錯誤，Snowplow 管道無法提交請求

Question

我在豐富步驟中收到錯誤“由於驗證錯誤而無法提交請求：INVALID_ARGUMENT：Pubsub 發布請求限制為 10MB，拒絕消息結束以避免超出 byte64 請求編碼的限制”。

我使用 Simo Ahava 的教程 在 Google Cloud Platform 上安裝 Snowplow | Simo Ahava 的博客

光束富集工作器的數據流步驟出現錯誤。 錯誤停止所有進程並且數據不會插入 BQ。

錯誤日志

{
“insertId”: “7514256621418980731:34459:0:179556”,
“jsonPayload”: {
“line”: “active_work_manager.cc:1564”,
“message”: “132593 Could not commit request due to validation error: INVALID_ARGUMENT: Pubsub publish requests are limited to 10MB, rejecting message over 7168K (size 7245K) to avoid exceeding limit with byte64 request encoding.”,
“thread”: “194”
},
“resource”: {
“type”: “dataflow_step”,
“labels”: {
“job_name”: “beam-enrich”,
“project_id”: “XXXXXXXXXX”,
“region”: “europe-central2”,
“job_id”: “2021-03-30_14_43_03-13952642482494084906”,
“step_id”: “”
}
},
“timestamp”: “2021-03-31T08:38:58.534286Z”,
“severity”: “ERROR”,
“labels”: {
“compute.googleapis.com/resource_name”: “beam-enrich-03301443-wkq8-harness-w1zs”,
“dataflow.googleapis.com/log_type”: “system”,
“dataflow.googleapis.com/job_id”: “2021-03-30_14_43_03-13952642482494084906”,
“dataflow.googleapis.com/region”: “europe-central2”,
“compute.googleapis.com/resource_type”: “instance”,
“compute.googleapis.com/resource_id”: “7514256621418980731”,
“dataflow.googleapis.com/job_name”: “beam-enrich”
},
“logName”: “projects/XXXXXXXXXX/logs/dataflow.googleapis.com%2Fshuffler”,
“receiveTimestamp”: “2021-03-31T08:39:21.828671489Z”
}

Answer 1

Pub/Sub 具有以下硬資源限制： https://cloud.google.com/pubsub/quotas#resource_limits

您的管道是否發布到 Pub/Sub？ 您可能需要通過拆分或截斷消息來減小消息的大小。