从Azure数据工厂访问Azure Blob存储帐户

Question

我的存储帐户中有一个包含文件列表的文件夹，并且一直在尝试使用管道删除其中一个文件。 为了完成该任务，我在管道中使用了“ Web”，复制了Blob存储URL和访问密钥。

使用Headers | Authorization下的访问键很累。 还尝试了https://docs.microsoft.com/zh-cn/azure/storage/common/storage-rest-api-auth#creating-the-authorization-header的共享密钥的概念

甚至尝试使用curl进行这项工作，但每次我尝试运行时都会返回“身份验证错误”

# List the blobs in an Azure storage container.

echo "usage: ${0##*/} <storage-account-name> <container-name> <access-key>"

storage_account="$1"
container_name="$2"
access_key="$3"

blob_store_url="blob.core.windows.net"
authorization="SharedKey"

request_method="DELETE"
request_date=$(TZ=GMT LC_ALL=en_US.utf8 date "+%a, %d %h %Y %H:%M:%S %Z")
#request_date="Mon, 18 Apr 2016 05:16:09 GMT"
storage_service_version="2018-03-28"

# HTTP Request headers
x_ms_date_h="x-ms-date:$request_date"
x_ms_version_h="x-ms-version:$storage_service_version"

# Build the signature string
canonicalized_headers="${x_ms_date_h}\n${x_ms_version_h}"
canonicalized_resource="/${storage_account}/${container_name}"

string_to_sign="${request_method}\n\n\n\n\n\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}\ncomp:list\nrestype:container"


# Decode the Base64 encoded access key, convert to Hex.
decoded_hex_key="$(echo -n $access_key | base64 -d -w0 | xxd -p -c256)"


# Create the HMAC signature for the Authorization header
signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary |  base64 -w0)

authorization_header="Authorization: $authorization $storage_account:$signature"

curl \
  -H "$x_ms_date_h" \
  -H "$x_ms_version_h" \
  -H "$authorization_header" \
  -H "Content-Length: 0"\
  -X DELETE  "https://${storage_account}.${blob_store_url}/${container_name}/myfile.csv_123"

curl命令返回错误：

<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:XX Time:2018-08-09T10:09:41.3394688Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request 'xxx' is not the same as any computed signature. Server used following string to sign: 'DELETE

Answer 1

您不能直接从数据工厂授权到存储帐户API。 我建议您使用逻辑应用程序。 Logic App内置了对Blob存储的支持： https : //docs.microsoft.com/zh-cn/azure/connectors/connectors-create-api-azureblobstorage

您可以从Data Factory Web Activity调用Logic App。 使用数据工厂请求的主体，您可以像blob路径一样将变量传递到Logic应用。

Answer 2

using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.Rest;
using Microsoft.Azure.Management.ResourceManager;
using Microsoft.Azure.Management.DataFactory;
using Microsoft.Azure.Management.DataFactory.Models;
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using Microsoft.WindowsAzure.Storage;

namespace ClearLanding
{
    class Program
    {
        static void Main(string[] args)
        {
            CloudStorageAccount backupStorageAccount = CloudStorageAccount.Parse("DefaultEndpointsProtocol=https;AccountName=yyy;AccountKey=xxx;EndpointSuffix=core.windows.net");
            var backupBlobClient = backupStorageAccount.CreateCloudBlobClient();
            var backupContainer = backupBlobClient.GetContainerReference("landing");
            var tgtBlobClient = backupStorageAccount.CreateCloudBlobClient();
            var tgtContainer = tgtBlobClient.GetContainerReference("backup");
            string[] folderNames = args[0].Split(new char[] { ',', ' ' }, StringSplitOptions.RemoveEmptyEntries);
            foreach (string folderName in folderNames)
            {
                var list = backupContainer.ListBlobs(prefix: folderName + "/", useFlatBlobListing: false);
                foreach (Microsoft.WindowsAzure.Storage.Blob.IListBlobItem item in list)
                {
                    if (item.GetType() == typeof(Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob))
                    {
                        Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob blob = (Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob)item;
                        if (!blob.Name.ToUpper().Contains("DO_NOT_DEL"))
                        {
                            var tgtBlob = tgtContainer.GetBlockBlobReference(blob.Name + "_" + DateTime.Now.ToString("yyyyMMddHHmmss"));
                            tgtBlob.StartCopy(blob);
                            blob.Delete();

                        }
                    }
                }
            }
        }
    }
}

我尝试通过编译上面的代码并使用C＃管道中的自定义活动引用它来解决此问题。 上面的代码段将文件从着陆文件夹传输到备份文件夹，并从着陆删除文件