BigQuery Execute 失敗，在 Cloud Data Fusion 上沒有有意義的錯誤

Question

我正在嘗試在 Cloud Data Fusion (Google) 中使用 BigQuery Execute function。 該組件驗證正常，SQL 檢查出來但每次執行時我都會收到這個無意義的錯誤：

02/11/2022 12:51:25 ERROR Pipeline 'test-bq-execute' failed.
02/11/2022 12:51:25 ERROR Workflow service 'workflow.default.test-bq-execute.DataPipelineWorkflow.<guid>' failed.
02/11/2022 12:51:25 ERROR Program DataPipelineWorkflow execution failed.

我看不到其他任何東西可以幫助我調試它。 有任何想法嗎？ 有問題的 SQL 是從 dataset.table WHERE ds = CURRENT_DATE() 中簡單刪除

這是管道

{
    "name": "test-bq-execute",
    "description": "Data Pipeline Application",
    "artifact": {
        "name": "cdap-data-pipeline",
        "version": "6.5.1",
        "scope": "SYSTEM"
    },
    "config": {
        "resources": {
            "memoryMB": 2048,
            "virtualCores": 1
        },
        "driverResources": {
            "memoryMB": 2048,
            "virtualCores": 1
        },
        "connections": [],
        "comments": [],
        "postActions": [],
        "properties": {},
        "processTimingEnabled": true,
        "stageLoggingEnabled": false,
        "stages": [
            {
                "name": "BigQuery Execute",
                "plugin": {
                    "name": "BigQueryExecute",
                    "type": "action",
                    "label": "BigQuery Execute",
                    "artifact": {
                        "name": "google-cloud",
                        "version": "0.18.1",
                        "scope": "SYSTEM"
                    },
                    "properties": {
                        "project": "auto-detect",
                        "sql": "DELETE FROM GCPQuickStart.account WHERE ds = CURRENT_DATE()",
                        "dialect": "standard",
                        "mode": "batch",
                        "dataset": "GCPQuickStart",
                        "table": "account",
                        "useCache": "false",
                        "location": "US",
                        "rowAsArguments": "false",
                        "serviceAccountType": "filePath",
                        "serviceFilePath": "auto-detect"
                    }
                },
                "outputSchema": [
                    {
                        "name": "etlSchemaBody",
                        "schema": ""
                    }
                ],
                "id": "BigQuery-Execute",
                "type": "action",
                "label": "BigQuery Execute",
                "icon": "fa-plug"
            }
        ],
        "schedule": "0 1 */1 * *",
        "engine": "spark",
        "numOfRecordsPreview": 100,
        "maxConcurrentRuns": 1
    }
}

Answer 1

我能夠使用 Cloud Logging 捕獲錯誤。 要在 Cloud Data Fusion 中啟用 Cloud Logging，您可以使用此GCP 文檔。 並按照以下步驟查看從 Data Fusion 到 Cloud Logging 的日志。 復制您的場景，這是我發現的錯誤：

      "logMessage": "Program DataPipelineWorkflow execution failed.\njava.util.concurrent.ExecutionException: com.google.cloud.bigquery.BigQueryException: Cannot set destination table in jobs with DML statements\n    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)\n    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)\n    at io.cdap.cdap.internal.app.runtime.distributed.AbstractProgramTwillRunnable.run(AbstractProgramTwillRunnable.java:274)\n    at org.apache.twill.interna..."
    }

我們為解決此錯誤所做的工作：無法使用 DML 語句在作業中設置目標表是我們在管道屬性中將Dataset Name和Table Name留空，因為不需要指定目標表。

Output：