需要实时分析方法

Question

My team wants to provide dashboards to users(store owners) with kpis specific to them, for eg orders placed in their stores, total orders cancelled, total sales, deliveries completed. etc. This dashboard should be updated near real time (say delay of 15 mins). It will be integrated with the core app where orders are placed.

One route we are assessing is to to replicate all required tables to a warehouse, lets say a db for analytics, join tables, keep the grain at transaction level and save this massive table to the same warehouse. All these steps should be done in 15 mins. The dashboard will then calculate kpis over this massive table. I think this is not the most efficient way.

My inclination is to not warehouse the tables and directly let the dashboards query on live tables. as per the current session, the filter at user level will automatically eliminate majority of records. So it would be fast enough.

Streaming tools like Spark streaming + Kafka may not help as i have to listen to many tables for updates and then aggregate.

Any suggestions on what could be the best approach? Please help

Answer 1

Doing analytics on your production transactional system is generally a bad idea (and why we have separate warehouse systems) because

1. It can affect the performance of the transactional processes
2. Data structures that effectively support OLTP tend to be very different to those that effectively support OLAP

As you would only load the deltas into your warehouse and calculate your KPIs against them, and not your whole dataset, then you shouldn't have performance issues. This is a pretty common pattern

But as David mentions, this is really just opinions as you don't provide enough information and are asking a very broad question