Azure Data Factory. Unable to load files via HTTP Connector

Question

To load files from a remote server to an Azure BLOB Storage, I am working on this tutorial from Azure team: Copy data from an HTTP endpoint by using Azure Data Factory or Azure Synapse Analytics .

I need to load publicly available files from this site through this API: Via API https://api.usaspending.gov/api/v2/download/disaster/

The tutorial for using the API (and other API from the above linked site) is provided here .

But when I run the pipeline, I get the following error:

ErrorCode=HttpFileFailedToRead,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read data from http server. Check the error from http server：The remote server returned an error: (405) Method Not Allowed.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (405) Method Not Allowed.,Source=System,'

Question : How can we use the the above API Endpoint in Azure Data Factory to load publicly available files from their remote server?

Remarks : According to their site : /api/v2/download/disaster/ POST Returns a zipped file containing Account and Award data for the Disaster Funding

Our configuration of the Source of ADF Pipeline :

The above source tests successfully as shown below :

The destination (an Azure Data Lake Storage) also tests successfully as shown below :

Answer 1

Check the METHOD which you are using in your copy activity. It needs to be POST.

Also, the response is not a file but provides a link to where the file can be downloaded. See below for the sample response. This means you need another activity to actually fetch the file.