数据流作业失败并且不显示任何日志

Question

I created pipelines in Dataflow using the standard template JDBC to BigQuery and there are a few jobs that are unexpectedly failing and not showing any logs.

The thing is, when a job fails because of the resources, the job needed more vCPUs than was avaliable in the region or the memory was not enough for example, these kind of errors are displayed in the logs, as you can see below.

But some jobs just fail with no logs and the resources are sufficient.

Does anyone know how to find the logs in this case?

Answer 1

Change the severity of the logs. If you choose Default , you should see more logs. For how the job page looks like for that failed job, I would say you are probably going to need to have a look at the worker logs as well.

Depending on the error, the Diagnostics tab may have some summarized info of what kind error has made the job fail.