[英]Data from HTTP endpoint to be loaded into Azure Data Lake using Azure Data Factory
I am trying to build a so called "modern data warehouse" using Azure services. 我正在尝试使用Azure服务构建一个所谓的“现代数据仓库”。
First step is to gather all the data in its native raw format into Azure Data Lake store. 第一步是将其本机原始格式的所有数据收集到Azure Data Lake存储中。 For some of the data sources we have no other choice than to use API for consuming the data. 对于某些数据源,我们别无选择,只能使用API来使用数据。 There's not much information when searching, therefore I am asking. 搜索时没有太多信息,所以我问。
Is it possible to define 2 Web Activities in my pipeline that will handle below scenario? 是否可以在我的管道中定义2个Web活动来处理下面的场景?
Above scenario is achievable by using Copy activity, but then I am not able to pass dynamic URL generated by Azure Functions. 通过使用Copy活动可以实现上述方案,但是我无法传递Azure Functions生成的动态URL。 How do I save the JSON output to ADL? 如何将JSON输出保存到ADL? Is there any other way? 还有其他方法吗?
Thanks! 谢谢!
Since you are using blob storage as an intermediary, and want to consume the blob upon creation, you could take advantage of Event Triggers . 由于您使用blob存储作为中介,并且想要在创建时使用blob,因此您可以利用事件触发器 。 You can set up the Event trigger to run a pipeline containing Web2 activity. 您可以设置事件触发器以运行包含Web2活动的管道。 Which kicks off when the Web1 activity completes (separate pipeline). 当Web1活动完成时(单独的管道)启动。
By separating the two activities into separate pipelines, the workflow becomes asynchronous. 通过将两个活动分成单独的管道,工作流变为异步。 This means you will not need to wait for both activities to complete before doing the next URL. 这意味着在执行下一个URL之前,您无需等待两个活动完成。 There are many other benefits as well. 还有许多其他好处。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.