Result of Bigquery job running on a table in which data is loaded via streamingAPI

Question

I have a BQ wildcard query that merges a couple of tables with the same schema (company_*) into a new, single table (all_companies). (all_companies will be exported later into Google Cloud Storage)

I'm running this query using the BQ CLI with all_companies as the destination table and this generates a BQ Job (runtime: 20mins+).

The company_* tables are populated constantly using the streamingAPI .

I've read about BigQuery jobs , but I can't find any information about streaming behavior.

If I start the BQ CLI query at T0, the streamingAPI adds data to company_* tables at T0+1min and the BQ CLI query finishes at T0+20min, will the data added at T0+1min be present in my destination table or not?

Answer 1

As described here the query engine will look at both the Columnar Storage and the streaming buffer, so potentially the query should see the streamed data.

It depends what you mean by a runtime of 20 minutes+. If the query is run 20 minutes after you create the job then all data in the streaming buffer by T0+20min will be included. If on the other hand the job starts immediately and takes 20 minutes to complete, you will only see data that is in the streaming buffer at the moment the table is queried.