Bigquery 在流式传输时无法加载所有数据

Question

Hi i am streaming data from my apps to Bigquery by C++. Thing got okay and the all are connectable, but the problem is the log file said there's 665 in streaming buffers enter image description here

However, final records in the table is just 4. Does anyone know to solve this? enter image description here

Answer 1

"Estimated rows" are only an estimate.

Streaming data in BigQuery is available in real-time (though table copy commands can take up to 90 minutes). I recommend reading this article for more information.

It sounds like you think you're losing data. That's not likely. I recommend checking what you believe is being inserted versus what's actually landing in the table.

Answer 2

Thank you for your great support. We found out that the issue happen because in the code we wrote, "insertId" should be unique, otherwise, BQ will deduplicate row with similar insertID. In the code, it happened since we set insertID = 'rowdetail' only. After changing it to time inserted (ddmmyyhhmmss), the pipeline run normally.

Hope it helpful for you and everyone