使用 Elastic Stack 对驻留在 Azure Data Lake Storage Gen2 中的数据进行实时数据分析

Question

How can we create the real-time data pipeline while data resides on Azure Data Lake Storage Gen2, and the analytics has to be done using Elastic Stack.

What can be the integration tool or technique for the completion of this design?

Answer 1

As @Nick.McDermaid mentioned in the comment that you need to reconsider your design. AFAIK there is no such tool available which can integrate Azure Data Lake Gen2 and Elastic Stack for real time analytics.

Alternatively, the better way to implement your requirement is by using the Azure products designed for real time analytics like Azure Stream Analytics , Azure Synapse Analytics, etc. You can also consider Azure Data Factory for data movement and transformation.

You can check outthis page to know more about all the analytics products available in Azure. Choose the best which suits your requirement and try to implement using official document examples.