如何使用Dataflow的jdbc to Bigquery模板将数据从Oracle 11g第2版传输到Bigquery？

Question

I have been looking for data transfer services to transfer data from Oracle 11g release 2 database. I came across some 3rd party solution which I tested worked perfectly.

But! I wanted to go with purely first party solution. So I looked into it and I came across Dataflow.

Dataflow already has some templates which I can use it to transfer the data.

I chose 'jdbc to bigquery' template.

It had some required parameters and optional paramters

Required: jdbc connection url, driver class name, source sql query, bigquery output table, GCS path for jdbc drivers, temporary directory for bigquery loading process, temporary location

Optional: jdbc Username, jdbc password

I setup a VM in compute engine to host a oracle 11g release 2 DB. I created a DB 'database', added a table 'table1', and some rows in it.

I created a job in dataflow with following paramters:

jdbc connection url: jdbc:oracle:thin:@VMinternalIP:1521:database

driver class name: oracle.jdbc.driver.OracleDriver

source sql query: select * from database.table1

bigquery output table: <testproject-0122>:<testdataset>.<table1>

GCS path for jdbc drivers: gs://testbucket/jdbc/ojdbc6.jar,gs://testbucket/jdbc/ojdbc5.jar,gs://testbucket/jdbc/ons.jar,gs://testbucket/jdbc/orai18n.jar,gs://testbucket/jdbc/simplefan.jar,gs://testbucket/jdbc/ucp.jar,gs://testbucket/jdbc/xdb6.jar

temporary directory for bigquery loading process: gs://testbucket/temporary

temporary location: gs://testbucket/temporary/amkortmp

jdbc username: username for oracle db

jdbc password: password for oracle db

However this job failed and throwed five errors.

Job status: Read from jdbcIO fail, Write to bigquery failed, drop inputs succeded

Four of five errors displayed the following error:

Edit: I updated my url and there is no connection error but I am getting the following error now:

ava.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist

I can see the table is in Oracle. So is it a schema error?

What am I doing wrong with this Dataflow template usage?

Answer 1

I eventually found the answer.

First the query is wrong on its reference to the table.

Correct format : select * from table1

Bigquery table reference is wrong

Correct format : project_ID:dataset_ID.table_ID

Correcting this, made it work.