BigQuery Java function 超时

Question

I am running a series of calculations on ~120 tables.

Algorithm: Read final table find where day=something and append the results to an existing table

    For table2 in tablearray
    {
    query = "SELECT * " +
    " FROM (SELECT * FROM tablefinal where day=someday) AS tmp " +
    "WHERE NOT EXISTS (" +
    "SELECT * FROM table2 t2 where  t2.user_id=tmp.user_id AND  t2.day=tmp.day);"
    
    then append rows to table2...
     QueryJobConfiguration queryConfig =
                    QueryJobConfiguration.newBuilder(query).setDestinationTable(destinationTable)
          .setWriteDisposition(JobInfo.WriteDisposition.WRITE_APPEND).build();
....

  bigquery.query(queryConfig);

    }

do that with 120 table still using finaltable as data source.

Issue. Is taking too long. Like 700 sec. And after 540 sec the google function dies. The results are correct, but it is too slow...

Question how can speed this up (The tables are pretty small ~100K rows)? Can I send a bunch of queries in parallel?

Answer 1

You can run the request concurrently in your code to parallelize the request but it's not the optimal way.

Personally, I recommend you to publish a PubSub message for each query to run. So, here the process:

• Cloud Scheduler runs a first Cloud Functions
• The first Cloud Functions iterates over the table to query and publish a message in PubSub for each query to perform: the parameters to include in the message are up to you
• A second Cloud Functions binds to the PubSub topic perform the query. (You can also create a PubSub push Subscription and call an HTTP Cloud FUnctions or something else in HTTP like Cloud Run or App Engine if you prefer)

The process is scalable (is tomorrow you have 250 tables, no worry, it scales.!). Don't forget to set the retry parameter on the second Cloud Functions because parallel requests at high rate to BigQuery can create transient issues

Answer 2

Try Magnus - Workflow Automator - you can find it on GCP Marketplace - as a part of Potens.io Suite of Tools for BigQuery

In Magnus you can easily build workflow with Loop Task that will take a list of tables and for each will execute whatever query/statement you need.

You can set Loop Task to be executed in Async Mode - so all tables will be process in parallel

As you can see you can even control how many concurrent iterations you allow - so you respect respective API limits

Check out https://potensio.zendesk.com for more details

Disclosure: I am author and lead on Potens.io Project by Viant Tech