使用 Google 的 Dataflow 时尝试使用 JDBC 到 BigQuery 模板时出现异常

Question

I'm trying to use Google's Dataflow using the JDBC to BigQuery template to extract information from Salesforce into BigQuery . The JDBC Driver I am using is Progress' DataDirect JDBC Driver for Salesforce .

I've confirmed that I can connect to Salesforce using the driver locally.

Unfortunately when I create a Job within Dataflow using the driver I get the following exception:

java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.sql.SQLException: Cannot get a connection, pool error Unable to activate object

I can provide a stack trace if necessary.

I've configured the template as follows (where values within <> are set appropriately):

Regional endpoint: us-central1
JDBC connection URL string: jdbc:datadirect:sforce://path.to.salesforce.com;SecurityToken=<Security Token>;User=<User>;Password=<Password>
JDBC driver class name: com.ddtek.jdbc.sforce.SForceDriver
JDBC source SQL query: SELECT Email FROM LEAD;
BigQuery output table: <project>:<dataset>.<table_name>
GCS paths for JDBC drivers: gs://<some_bucket>/sforce.jar
Temporary directory for BigQuery loading process: gs://<some_bucket>/staging
Temporary location: gs://<some_bucket>/tmp

I'm hoping that someone familiar with this use case will be able to help me determine why I'm getting this exception and how I can go about resolving it. My working assumptions are that I'm using the driver incorrectly, I need to add additional drivers to GCS, or there is an issue connecting to Salesforce through Dataflow.

Edit:

The JDBC driver from Progress that I am using is version 6.0. I'm unsure which version of the Dataflow SDK is being used when I use the UI but when I have tried working with the Google Dataflow SDK locally it looks like I'm using version 2.5.0 based on the pom.xml.

Answer 1

I didn't find any relevant issue between the JDBC and Dataflow versions, so I will suggest to take a look on the following tutorials from Progress website to understand if you are using the driver correctly:

1. Setting Google Cloud Dataflow SDK
2. Import data from Salesforce into Google Bigquery using Dataflow

and if the exception persist, please create an issue with GCP following this guide