BigQuery：写后读一致性（流式插入）

Question

As the title states, what is the consistency guarantee for BigQuery with respect to streaming inserts? Will I be able to guarantee a consistent read after inserting rows using streaming insert?

Answer 1

Yes and no!!

• Yes, the read is consistent if you get your data just after the streaming insert!
• No if you use partitioning because the streaming data are stored in a buffer and it can take up to 2h to BigQuery to store the data in the correct partition.

So, it depends on how you use your data.

Answer 2

It depends on which Streaming API you are using and which DML you expect to run:

• When using the Legacy Streaming API, " Data is available for real-time analysis using standard SQL queries immediately after BigQuery successfully acknowledges a tabledata.insertAll request "
• Using the new Storage Write API, the behaviour is configurable: You can choose between default stream, committed mode, pending mode, and buffered mode, depending on your requirements :
  • Committed mode and default stream : "Records are available for reading immediately as you write them to the stream". The difference is that the default stream comes without exactly-once guarantees and, consequently, potentially higher throughputs.
  • Pending mode : "Records are buffered in a pending state until you commit the stream. Once a stream is committed, all of the data written to the stream is available for reading"
  • Buffered mode : "Records are buffered until you flush the stream. Flushing advances a cursor to a specified offset and makes all the records up to that offset visible"
• Nevertheless, " rows that were written to a table recently by using streaming (the tabledata.insertall method or the Storage Write API) cannot be modified with UPDATE, DELETE, or MERGE statements " and " Some recently streamed rows might not be available for table copy typically for a few minutes. In rare cases, this can take up to 90 minutes "
• " Because BigQuery's streaming API is designed for high insertion rates, modifications to the underlying table metadata are eventually consistent when interacting with the streaming system "

So in general terms, when using default streaming, data is promptly available for reading queries after an acknowledged streaming insert.