從 blobstorage 獲取資源文件並添加到 azure 批處理中的任務

Question

我是 Azure 和 Azure Batch 服務的新手。 我創建了一個簡單的 .NET 應用程序，它應該在某個輸入文件上運行一些代碼。 這就是我開始創建作業、任務並添加資源文件的方式，這些資源文件應該由我的應用程序包處理。

1.) 創建 BlobClient 等：

        // Create the blob client, for use in obtaining references to blob storage containers
        CloudBlobClient blobClient = CreateCloudBlobClient(StorageAccountName, StorageAccountKey);

        // Use the blob client to create the input container in Azure Storage 
        const string inputContainerName = "modelinput";

        CloudBlobContainer container = blobClient.GetContainerReference(inputContainerName);

        container.CreateIfNotExistsAsync().Wait();

2.) 這里一個放在應用程序目錄中的文件被上傳並添加到資源文件列表中：

         List<ResourceFile> inputFiles = new List<ResourceFile>();

//upload the file that should be processed and add to resourcefiles
inputFiles.Add(UploadFileToContainer(blobClient, inputContainerName, "myinputfile.xml"))

3.) 創建批處理作業和任務

    BatchSharedKeyCredentials cred = new BatchSharedKeyCredentials(BatchAccountUrl, BatchAccountName, BatchAccountKey);

        using (BatchClient batchClient = BatchClient.Open(cred))
        {

            Console.WriteLine("Creating job [{0}]...", JobId);


                CloudJob job = batchClient.JobOperations.CreateJob();
                job.Id = JobId;
                job.PoolInformation = new PoolInformation { PoolId = PoolId }; 

                job.Commit();

        List<CloudTask> tasks = new List<CloudTask>();

        string taskId = String.Format("Task{0}", 0);
        string inputFilename = inputFiles[0].FilePath;

        //set the commandline 
        string taskCommandLine = String.Format("cmd /c %AZ_BATCH_APP_PACKAGE_DEVBATCHAPP%\\batchapp.exe {0}", inputFilename);

        CloudTask task = new CloudTask(taskId, taskCommandLine)

         //add my resourcefiles to the task
        task.ResourceFiles = new List<ResourceFile> { inputFiles[0] }; 

        task.ApplicationPackageReferences = new List<ApplicationPackageReference>{new ApplicationPackageReference{ApplicationId = "devbatchapp",Version = "0.0.1"}};

        tasks.Add(task);
     }

如果我現在運行任務，一切都會完美運行。 但是現在我開發了一個小的 ASP.NET Razor Pages 應用程序，用戶應該通過它選擇和上傳文件。 由於我現在只能使用文件流來上傳我的文件，因此我必須將 2.) 更改為主要遵循此處的文檔：

            var filePath = Path.GetTempFileName();

            var file = Path.Combine(_environment.ContentRootPath, "uploads", filePath); 

        string containerSasToken = container.GetSharedAccessSignature(new SharedAccessBlobPolicy()
        {
            SharedAccessStartTime = DateTime.UtcNow,
            SharedAccessExpiryTime = DateTimeOffset.UtcNow.AddDays(1),
            Permissions = SharedAccessBlobPermissions.Write | SharedAccessBlobPermissions.Read
        });
        string containerSas = container.Uri.AbsoluteUri + containerSasToken; 

        using (var stream = new FileStream(file, FileMode.Create))
        {
            await Upload.CopyToAsync(stream);
            stream.Seek(0, SeekOrigin.Begin);
            var blobstore = container.GetBlockBlobReference("modelinput");
            await blobstore.UploadFromStreamAsync(stream);

            containerSas = blobstore.Uri.AbsoluteUri + containerSasToken;
            inputFiles = new List<ResourceFile> { 
            ResourceFile.FromStorageContainerUrl(containerSas, "myinput") };

        }

其余代碼基本保持不變。 但是現在當我嘗試運行任務時出現BlobDownloadMiscError 。 在 Batch explorer 中分析這個時，我發現資源文件的 URL 顯然是錯誤的，如果我手動將它添加到它工作的任務中。 有人可以幫助我嗎？ 如何獲取資源文件的正確來源以將其添加到我的任務中？

Answer 1

根據我的測試，您可能沒有使用正確的權限。 更多詳細信息，請參閱文檔。 此外，請確保您的存儲帳戶已鏈接到您的批處理帳戶。

對於容器訪問，您必須同時擁有讀取和列表權限，而對於 blob 訪問，您只需要讀取權限。

我的代碼如下

string containerSasToken = container.GetSharedAccessSignature(new SharedAccessBlobPolicy()
{
    SharedAccessStartTime = DateTime.UtcNow,
    SharedAccessExpiryTime = DateTimeOffset.UtcNow.AddDays(1),
    Permissions = SharedAccessBlobPermissions.Write | SharedAccessBlobPermissions.Read | SharedAccessBlobPermissions.List
});

string containerSasUrl = String.Format("{0}{1}", container.Uri, containerSasToken);
var inputFiles = new List<ResourceFile> { };
var file = ResourceFile.FromStorageContainerUrl(containerSasUrl,"test");
inputFiles.Add(file);
Console.WriteLine(inputFiles.Count);

// Get a Batch client using account creds

BatchSharedKeyCredentials cred = new BatchSharedKeyCredentials(BatchAccountUrl, BatchAccountName, BatchAccountKey);

using (BatchClient batchClient = BatchClient.Open(cred))
{
    Console.WriteLine("getting pool [{0}]...", PoolId);


    batchClient.PoolOperations.GetPool(PoolId);
    
    // Create a Batch job
    Console.WriteLine("Creating job [{0}]...", JobId);

    try
    {
        CloudJob job = batchClient.JobOperations.CreateJob();
        job.Id = JobId;
        job.PoolInformation = new PoolInformation { PoolId = PoolId };

        job.Commit();
    }
    catch (BatchException be)
    {
        // Accept the specific error code JobExists as that is expected if the job already exists
        if (be.RequestInformation?.BatchError?.Code == BatchErrorCodeStrings.JobExists)
        {
            Console.WriteLine("The job {0} already existed when we tried to create it", JobId);
        }
        else
        {
            throw; // Any other exception is unexpected
        }
    }

    // Create a collection to hold the tasks that we'll be adding to the job

    Console.WriteLine("Adding {0} tasks to job [{1}]...", inputFiles.Count, JobId);

    List<CloudTask> tasks = new List<CloudTask>();

    // Create each of the tasks to process one of the input files. 

    for (int i = 0; i < inputFiles.Count; i++)
    {
        string taskId = String.Format("Task{0}", i);
        string inputFilename = inputFiles[i].FilePath;
        string taskCommandLine = String.Format("cmd /c type {0}", inputFilename);

        CloudTask task = new CloudTask(taskId, taskCommandLine);
        task.ResourceFiles = new List<ResourceFile> { inputFiles[i] };
        tasks.Add(task);
    }

    // Add all tasks to the job.
    batchClient.JobOperations.AddTask(JobId, tasks);


    // Monitor task success/failure, specifying a maximum amount of time to wait for the tasks to complete.

    TimeSpan timeout = TimeSpan.FromMinutes(30);
    Console.WriteLine("Monitoring all tasks for 'Completed' state, timeout in {0}...", timeout);

    IEnumerable<CloudTask> addedTasks = batchClient.JobOperations.ListTasks(JobId);

    batchClient.Utilities.CreateTaskStateMonitor().WaitAll(addedTasks, TaskState.Completed, timeout);

    Console.WriteLine("All tasks reached state Completed.");

    // Print task output
    Console.WriteLine();
    Console.WriteLine("Printing task output...");
}