BigQuery streaming inserts fail on GKE

Question

We have GKE cluster with 3x n2-highcpu-8 nodes, web application written in GO scaled to 3 instances (1 per each node) that writes all requests using streaming to BigQuery, and I noticed quite weird behaviour:

During high application usage 2 out of 3 instances of application starts to 100% fail in streaming writes with writing only "context deadline exceeded” as an error, and when I delete those 2 pods and they are getting back to receiving traffic the old 1 starts failing with "context deadline exceeded”, while 1 of new 2's are successfully continue writing data and another starts failing.

I went through quotes and limits available on BigQuery documentation and haven't found anything that might be related to this case, looking into Stackdriver Monitoring to see how many writes there are per table per second and numbers around 1500, and the size of data that sent is also quite small 1-5kb, we don't use batch writes so it is mostly done through goroutines ASAP as request comes.

BigQuery Logging don't have any errors/warnings as well.

Is there any hidden limitation, or overall BigQuery streaming writes only good for some small amount of simultaneous writes and we need some queue solution using Pub/Sub and Dataflow to transport data to BigQuery in high volume?

GKE and BigQuery dataset is located in europe-west-2 and this happens every day

[EDIT]

Here is some streaming statistics from one of the biggest tables if it does make any difference:

Streaming buffer statistics Estimated size 249.57 MB Estimated rows 1,640,220 Earliest entry time 3 Dec 2020, 18:43:00

Answer 1

Actually the issue was related to misconfiguration of Affinity settings of the application, and 2 pods been running on same GKE node, which during primetime consumed 100% of cpu and that seems to be an related issue, so after that was sorted we haven't seen any context deadlines messages or errors with writing to BigQuery