How to get individual row from bigquery table less then a second?

Question

I have a aggregated data.table in bigquery that has millions of rows. This table is growing everyday.

I need a way to get 1 row from this aggregate table in milliseconds to append data in real time event.

What is the best way to tackle this problem?

Answer 1

BigQuery is not build to respond in miliseconds, so you need an other solution in between. It is perfectly fine to use BigQuery to do the large aggregration calculation. But you should never serve directly from BQ where response time is an issue of miliseconds.

Also be aware, that, if this is an web application for example, many reloads of a page, could cost you lots of money.. as you pay per Query.

There are many architectual solution to fix such issues, but what you should use is hard to tell without any project context and objectives.

For realtime data we often use PubSub to connect somewhere in between, but that might be an issue if the (near) realtime demand is an aggregrate.

You could also use materialized views concept, by exporting the aggregrated data to a sub component. For example cloud storage -> pubsub, or a SQL Instance / Memory store.. or any other kind of microservice.