如何提高 BigQuery 读取性能

Question

We're using BigQuery to retrieve the full content of a big table. We're using the publicly available publicdata:samples.natality.

Our code follows Google instructions as described in their API doc - java .

We're able to retrieve this table at around 1'300 rows/sec that is amazingly slow. Is there a faster way to retrieve the full result of a query or is this always as fast as it gets ?

Answer 1

The recommended way to retrieve a large amount of data from a BigQuery table is not to use tabledata.list to page through a full table as that example is using. That example is optimized for reading a small number of rows for the results of a query.

Instead, you should run an extract job that exports the entire content of the table to Google Cloud Storage, which you can then download the full content from.

https://cloud.google.com/bigquery/exporting-data-from-bigquery

Answer 2

To download a table fast you can use Google BigQuery Storage Client for Java.

It lets you download the tables into efficient binaries format such as Avro or Arrow. Using the basic Arrow example in the documentation I manage to download ~1 million rows per second.

I think you can use it to download a query result by writing the result into a temporary table.

The code to get the temporary table of the result looks like this:

public static TableId getTemporaryTable(String query) throws InterruptedException{
    QueryJobConfiguration queryConfig = 
                QueryJobConfiguration.newBuilder(query)
                    .setUseLegacySql(false)
                    .build();
    Job queryJob = bigquery.create(JobInfo.newBuilder(queryConfig).build());
    queryJob = queryJob.waitFor(); // Wait for the query to complete.
    return ((QueryJobConfiguration) queryJob.getConfiguration()).getDestinationTable();
}

References:

• Google cloud documentation
• GitHub repository