将BigQuery中的许多Firebase数据集聚合到单个数据集中

Question

For starters, I am relatively new to Firebase and BigQuery...so a bit naive...

---

Background :

I am working on a project where we have many distinct mobile apps - one for each country, for each OS (iOS, Android) - with all of the raw event data getting pushed into BigQuery datasets that live in the same project. The rationale behind having distinct apps by country relates to regulatory requirements.

For dashboarding purposes, I had hoped to be able to combine all datasets into a single aggregate dataset (with tables by year) that is partitioned by date via a "partition_date" column that I added. My goal has been to take advantage of the ability to use BigQuery nested structures, and to maintain the original nested fields from the raw Firebase tables.

I've been searching around over the past month or so, but I haven't come across any other use cases where people are dealing with many datasets.

---

Questions :

• Does it make sense to take this approach (aggregate all datasets into a table partitioned by date, and a nested field based on app name)?
• As an alternative - I tried denormalizing the tables, with tables sharded by date...though the resulting tables (not surprisingly) are much larger due to flattening the nested structures, which makes me think this approach is less than ideal. Should I actually consider this approach?
• Should I consider just making a relatively simple aggregate table of KPIs? I've avoided this approach because it limits my ability to use date range filters that dynamically calculate unique users on the fly (using COUNT_DISTINCT(user_dim.app_info.app_instance_id) in Data Studio)

---

The end goal is to empower any user (ex. someone who doesn't know SQL) to be able to answer simple questions across all apps (ex. how many users opened the app yesterday), and to let end users make use of date range filters in the dashboard interface.

I've been able to write ad hoc queries to get at the answers by querying across all datasets, but I have not found a good solution that will make it easier for non-technical users within the dashboard.

Also, this is my first Stack Overflow question...please let me know if I am being too vague, including too many questions, or otherwise abusing the platform.

Thanks in advance for any thoughts.

Answer 1

If your end goal is to empower users to answer simple questions, aggregating KPI data makes sense to me. I would probably approach this by setting up a scheduled daily job that selects the relevant data from all the datasets and loads it into a new dataset, which can then be used in Data Studio. The new datasets could have the default Firebase date table suffix to support date range filters.

I am also relatively new to BigQuery and Firebase though, so maybe there is a better way.

You can find more information about scheduling in BigQuery here: Schedule query in BigQuery