谷歌云数据流 JDBC 到 BigQuery 作业只是挂起并且没有产生有用的日志/错误消息

Question

I am trying to use the Google provided Dataflow template JDBC to BigQuery. I have an on-prem SQL Server that I want to connect to. I have configured all the necessary networking and firewall rules to allow the Dataflow workers to communicate with the on-prem server. I have also confirmed that the provided connection string and credentials work.

This issue is that when I run the job, it just hangs and doesn't produce any helpful log/errors. I have check the logs explorer and can't seem to find anything to help troubleshoot this.

If I let it run, eventually it hits a timeout and produces this: useless log

Answer 1

I would suggest opening a support ticket, as it may be necessary to have the Job ID to see exactly what went wrong.

Such symptom can be observed when created workers can't be reached by Dataflow -- either by having the subnetwork mismatching the job region, or because of strict firewall rules.

Answer 2

Usually when I have dataflow job hanging, I open the full log query and I keep only the job id in the log query.

The log query by default is:

resource.type="dataflow_step"
resource.labels.job_id="job_id"
logName=("projects/project/logs/dataflow.googleapis.com%2Fjob-message" OR "projects/project/logs/dataflow.googleapis.com%2Flauncher")

I just change it to:

resource.labels.job_id="job_id"

This usually gives me more details about the pipeline and what's wrong. It happened with me a lot when dealing with customer containers and flex templates etc.