Data analytics (join mongoDB and SQL data) through Azure Data Lake and power BI

Question

We have an app hosted on Azure using mongoDB (running on a VM) and Azure SQL dbs. The idea is to build a basic data analysis pipeline to "join" the data between both these DBs and visually display the same using power BI.

For instance we have a "user" table in SQL with a unique "id" and a "data" table in mongo that has a reference of "id" + other tables in SQL that have reference of 'id'. So we wish to analyse the contents of data based on user and possibly join that further with other tables as needed.

Is azure data lake + power BI enough to implement this case? Or we need azure data analytics or azure synapse for this?

Answer 1

Azure Data Lake (ADL) and Power BI on its own is not going to be able to build a pipeline, ADL it is just a storage area and Power BI is a very much a lightweight ETL tool limited by features and capacity.

It would be highly recommended that you have some better compute power behind it using, as you mentioned Azure Synapse. That will be able to have a defined pipeline to orchestrate data movement into the data lake, then do the processing to transform the data.

Power BI on it own will not be able to do this, as you will still be limited by the Dataflow and Dataset size of 1GB if running Pro. Azure Synapse does contain Azure Data Factory Pipelines, Apache Spark and Azure SQL Data Warehouse so you can choose between Spark and SQL for your data transformational steps as both will connect to the Data Lake.

Note: Azure Data Lake Analytics (ADLA) (and USQL) is not a major focus for MS, and never widely used. Azure Databricks and Azure Synapse with Spark has replaced ADLA in all of the modern data pipeline and architectures examples for MS.