Google Cloud Platform Data Display

Question

I currently have an app running on Google Cloud Platform that is collecting a decently large amount of data and storing it in Google Cloud Datastore, and I am displaying this data in plots using Google Data Studio. However, in order to do so, I have to send backups of the data to Google Cloud Storage, and then load these backups into Google BiqQuery, which can then be plotted on Google Data Studio. Since my app is collecting live data in real time, this is a less-than-ideal solution, as I cannot view the live data on my plots: only the data I have already backed up and loaded. I was wondering if there is a better way to do this so that I could show the live data directly in Google Data Studio? The alternative that I'm considering is writing a front-end page for my app and doing this all manually in Python, which I'd prefer to avoid if possible.

Any advice is much appreciated!

Answer 1

I'd investigate pushing data from the app directly into BiqQuery. From Streaming Data into BigQuery :

Instead of using a job to load data into BigQuery , you can choose to stream your data into BigQuery one record at a time by using the tabledata().insertAll() method. This approach enables querying data without the delay of running a load job. This document discusses several important trade-offs to consider before choosing an approach, including streaming quotas, data availability, and data consistency.