Data Ingestion Patterns in Data Factory using REST API

Question

I am reaching out to you gather best practices around ingestion of data from various possible API's into a Blob Storage. I am considering to interface with all the various possible API's using Data Factory and here are the possible set of API's which I have currently:

1. Ingesting from an API service within the same Resource Group in Azure Cloud

2. Ingesting from an API service from a different resource group but within the same Azure subscription

3. Ingestion from an API service from a different Azure Subscription / VNET

4. Ingestion from an API service available publicly such as Twitter, Facebook

5. Ingestion from an API service which is available on-premises

6. Any other possible API services

My questions around the above API services are:

a) What are the specific security related settings I need to take care in order to interface with the above API's (Managed Identity, Service Principal etc.)

b) When to use which security setting ?

c) Along with Azure Data Factory, is there any other Azure service which can be leveraged for the above ingestion from the API's

d) What are the specifics of Runtimes / Linked services which I should be taking care about e) Any specifics around AAD resource and Authentication type

Answer 1

Just wanted to add that for #5 , you will have to use SHIR ( Self hosted IR ) . Please read about this here .

Answer 2

for c) Along with Azure Data Factory, is there any other Azure service which can be leveraged for the above ingestion from the API's

Logic Apps has connectors backed in for many existing API's