java.lang.NoSuchMethodError: BigQuery Java API throws com.google.api.services.bigquery.model.JobStatistics2.getDdlTargetRoutine() not found

Question

I'm trying to run the BigQuery APIs in a Dataflow job to execute a custom query against BigQuery. I tried to follow the instructions from here . But every time I'm running the job I get the below error:

{ 
   "@timestamp":"2019-11-20T16:12:51.008Z",
   "severity":"ERROR",
   "pid":"40364",
   "thread":"main",
   "class":"o.s.b.d.LoggingFailureAnalysisReporter",
   "msg":"\r\n\r\n
***************************\r\nAPPLICATION FAILED TO START\r\n***************************\r\n
    Description:  

    An attempt was made to call a method that does not exist. The attempt was made from the following location:
        com.google.cloud.bigquery.JobStatistics$Builder.<init>(JobStatistics.java:1066)

        The following method did not exist:
com.google.api.services.bigquery.model.JobStatistics.getNumChildJobs()Ljava/lang/Long;

        The method's class, com.google.api.services.bigquery.model.JobStatistics, is available from the following locations:
            jar:file:/C:/Users/{user}/.m2/repository/com/google/apis/google-api-services-bigquery/v2-rev20181104-1.27.0/google-api-services-bigquery-v2-rev20181104-1.27.0.jar!/com/google/api/services/bigquery/model/JobStatistics.class

        It was loaded from the following location:
            file:/C:/Users/{user}/.m2/repository/com/google/apis/google-api-services-bigquery/v2-rev20181104-1.27.0/google-api-services-bigquery-v2-rev20181104-1.27.0.jar

    Action:
        Correct the classpath of your application so that it contains a single, compatible version of com.google.api.services.bigquery.model.JobStatistics"
}

Here are the dependencies added:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>libraries-bom</artifactId>
            <version>2.8.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-bigquery</artifactId>
</dependency>

And this is the code to execute the query and issue seems to be in the bigquery.create(JobInfo.newBuilder(queryConfig).setJobId(jobId).build()) line.

BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();
QueryJobConfiguration queryConfig = QueryJobConfiguration
    .newBuilder(query)
    .setUseLegacySql(false).build();
// Create a job ID so that we can safely retry.
JobId jobId = JobId.of(UUID.randomUUID().toString());
System.out.println("created the job id");

Job queryJob = bigquery.create(
    JobInfo.newBuilder(queryConfig).setJobId(jobId).build());
queryJob = queryJob.waitFor();

Answer 1

From the original post, I understand that you want to request data from BigQuery to use it as input on a Dataflow Pipeline.
Instead of using the the BigQuery API, I would suggest you to use the Built-in I/O transforms from Apache Beam, specifically the BigQuery connector .

Answer 2

I had the same issue. Got resolved after upgrading google-cloud-bigquery to 1.116.0

<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-bigquery</artifactId>
    <version>1.116.0</version>
</dependency>