Exception when trying to use JDBC to BigQuery template when using Google's Dataflow

Question

I'm trying to use Google's Dataflow using the JDBC to BigQuery template to extract information from Salesforce into BigQuery . The JDBC Driver I am using is Progress' DataDirect JDBC Driver for Salesforce .

I've confirmed that I can connect to Salesforce using the driver locally.

Unfortunately when I create a Job within Dataflow using the driver I get the following exception:

java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.sql.SQLException: Cannot get a connection, pool error Unable to activate object

I can provide a stack trace if necessary.

I've configured the template as follows (where values within <> are set appropriately):

Regional endpoint: us-central1
JDBC connection URL string: jdbc:datadirect:sforce://path.to.salesforce.com;SecurityToken=<Security Token>;User=<User>;Password=<Password>
JDBC driver class name: com.ddtek.jdbc.sforce.SForceDriver
JDBC source SQL query: SELECT Email FROM LEAD;
BigQuery output table: <project>:<dataset>.<table_name>
GCS paths for JDBC drivers: gs://<some_bucket>/sforce.jar
Temporary directory for BigQuery loading process: gs://<some_bucket>/staging
Temporary location: gs://<some_bucket>/tmp

I'm hoping that someone familiar with this use case will be able to help me determine why I'm getting this exception and how I can go about resolving it. My working assumptions are that I'm using the driver incorrectly, I need to add additional drivers to GCS, or there is an issue connecting to Salesforce through Dataflow.

Edit:

The JDBC driver from Progress that I am using is version 6.0. I'm unsure which version of the Dataflow SDK is being used when I use the UI but when I have tried working with the Google Dataflow SDK locally it looks like I'm using version 2.5.0 based on the pom.xml.

Answer 1

I didn't find any relevant issue between the JDBC and Dataflow versions, so I will suggest to take a look on the following tutorials from Progress website to understand if you are using the driver correctly:

1. Setting Google Cloud Dataflow SDK
2. Import data from Salesforce into Google Bigquery using Dataflow

and if the exception persist, please create an issue with GCP following this guide