403 error when trying to access file system in Azure data lake storage Gen 2 via REST API

Question

I am trying to access file system in azure data lake storage gen 2 via REST API using java. this is how I am building my request:

public static void main(String[] args) throws Exception {
    String urlString = "https://" + account + ".dfs.core.windows.net/sterisfiles?resource=filesystem";
    HttpURLConnection connection = (HttpURLConnection)(new URL(urlString)).openConnection();
    getFileRequest(connection, account, key);
    connection.connect();
    System.out.println("Response message : "+connection.getResponseMessage());
}


public static void getFileRequest(HttpURLConnection request, String account, String key) throws Exception{
    SimpleDateFormat fmt = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss");
    fmt.setTimeZone(TimeZone.getTimeZone("GMT"));
    String date = fmt.format(Calendar.getInstance().getTime()) + " GMT";
    String stringToSign =  "GET\n"
            + "\n" // content encoding
            + "\n" // content language
            + "\n" // content length
            + "\n" // content md5
            + "\n" // content type
            + "\n" // date
            + "\n" // if modified since
            + "\n" // if match
            + "\n" // if none match
            + "\n" // if unmodified since
            + "\n" // range
            + "x-ms-date:" + date + "\n"
            + "x-ms-version:2014-02-14\n" //headers
            + "/"+account + request.getURL().getPath();
    String auth = getAuthenticationString(stringToSign);
    request.setRequestMethod("GET");
    request.setRequestProperty("x-ms-date", date);
    request.setRequestProperty("x-ms-version", "2014-02-14");
    request.setRequestProperty("Authorization", auth);
}

private static String getAuthenticationString(String stringToSign) throws Exception{
    Base64 base64 = new Base64();
    Mac mac = Mac.getInstance("HmacSHA256");
    mac.init(new SecretKeySpec(base64.decode(key), "HmacSHA256"));
    String authKey = new String(base64.encode(mac.doFinal(stringToSign.getBytes("UTF-8"))));
    String auth = "SharedKey " + account + ":" + authKey;
    return auth;
}

This is throwing 403 error with message: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

are my request headers not correct?

Answer 1

According to my test, we can use Azure AD authentication to call Azure data lake storage Gen2 REST API. For more details, please refer to https://social.msdn.microsoft.com/Forums/en-US/45be0931-379d-4252-9d20-164261cc64c5/error-while-calling-adls-gen-2-rest-api-to-create-file?forum=AzureDataLake .

1. Create Azure AD service principal and assign a RABC role to it. For futher information, please refer to https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad .

az ad sp create-for-rbac -n 'your sp name' --role 'Storage Blob Data Owner' --scope 'your scope such as your storage account scope'

2. Get access token

Method : POST 
URL: https://login.microsoftonline.com/<your Azure AD tenant domain>/oauth2/token
Body:
     grant_type =client_credentials 
    client_id=<the appid you copy>
    client_secret=<the password you copy>
    resource=https://storage.azure.com

3. Call rest api a. Create File system

   PUT https://{accountName}.{dnsSuffix}/{filesystem}?resource=filesystem

   

   b. List File system

   GET https://{accountName}.{dnsSuffix}/?resource=account