Azure Data Lake：向Azure Data Lake Store的请求未经授权

Question

We are trying to move our data from Azure Table Storage to Azure Data Lake. So we created a Data Factory (with linked services, data sets and pipeline). Pipline is created with Copy Action.

One of the linked service we choose "Azure Data Lake Store" and we authorized configuration and the finally configuration is like below

{
    "name": "XXXXXStoreLinkedService",
    "properties": {
        "description": "",
        "hubName": "XXXXXXdatafactory_hub",
        "type": "AzureDataLakeStore",
        "typeProperties": {
            "dataLakeStoreUri": "https://XXXXXX.azuredatalakestore.net/webhdfs/v1",
            "authorization": "**********",
            "sessionId": "**********",
            "subscriptionId": "XXXXXXXXXXXXXXXXXXX",
            "resourceGroupName": "XXXXXXXXXXXXXXX"
        }
    }
}

After creating the pipeline , it seems that the data movement is happening. But we are getting this error

Copy activity encountered a user error: ErrorCode=UserErrorAdlsUnauthorizedAccess,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The request to 'Azure Data Lake Store' was unauthorized,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (403) Forbidden.,Source=System,'

From this error, it seems that we have an unauthorized request. But as mentioned we above we authorized the Azure Data Lake store linked service correctly.

Can anybody please let us know, what are the possible cases of this error and also what we need to do here.

Answer 1

Can you please confirm if the user you are getting the authorization information for, has been allowed access to the appropriate Data Lake Store account?

https://azure.microsoft.com/en-us/documentation/articles/data-lake-store-secure-data/

Details are available at the above link. It is 3-step process. So, do ensure you have completed them all.

Thanks, Sachin

Program Manager, Azure Data Lake

Answer 2

The way we got around this was by opening up the access by checking the three boxes next to "All Users and Groups":

I created all the data factory and data lake store assets as the Owner and still got the authorization error that you did. The only difference is that my Data Factory is on a different subscription than the Data Lake Store. I don't think that this should matter since I am an admin on both subscriptions with the same Microsoft login and am an owner on the Data Lake Store...There is probably a more focused way to do this but the AAD stuff referenced in the accepted answer didn't work for me...until then this will work.

Answer 3

You need to set recursive execute permissions at a root level in order to perform operations on files below the directory structure. This is easier explained in an image. Please see -

https://azure.microsoft.com/en-gb/documentation/articles/data-lake-store-access-control/#common-scenarios-related-to-permissions