带有KMS的AWS S3客户端加密在最后转移的部分上发出ProtocolViolationException

Question

The following code attempts to copy a 17MB test file to an S3 bucket using multi-part transfer, client-side envelope encryption and the Amazon KMS service to handle the data encryption key. The multi-part block size is 5MB.

Upon transferring the last (partial) block, and only if the IsLastPart flag is set to true , the call to UploadPart generates a System.Net.ProtocolViolationException indicating: Bytes to be written to the stream exceed the Content-Length bytes size specified.

This suggests that the Content-Length html header was not updated to reflect the necessary "pad bytes" added by the encryption engine to the last cipher block for proper alignment. As a result, when those final bytes were added, they exceeded the given Content-Length and generated this error.

If IsLastPart is not set (ie left false ), then the operation succeeds, but upon downloading and decrypting the operation also fails.

NOTE: The KmsAlgorithm class is not provided by the AWS .NET SDK. This class comes from another Stack Overflow posting because the .NET version of the AWS SDK does not provide a connector class between KMS and S3 to support envelope encryption as the Java SDK does.

So what is the proper way to send multi-part uploads to S3 with client-side encryption and KMS managed keys?

    static string bucketName = "*****************************";
    static string keyName = "test.encrypted.bin";
    static string uploadSourcePath = "c:\\temp\\test.bin";
    static long partSize = 5 * 1024 * 1024;
    static String uploadId = "";

    static void Main(string[] args)
    {
        if (checkRequiredFields())
        {
            String cmkId = "************************************";

            // Prepare our KMS client and kmsAlgorithm
            using (AmazonKeyManagementServiceClient kmsClient = new AmazonKeyManagementServiceClient())
            using (KMSAlgorithm kmsAlgo = new KMSAlgorithm(kmsClient, cmkId))
            {
                // Generate the encryption materials object with the algorithm object
                EncryptionMaterials encryptionMaterials = new EncryptionMaterials(kmsAlgo);

                // Now prepare an S3 crypto client
                using (AmazonS3EncryptionClient cryptoClient = new AmazonS3EncryptionClient(encryptionMaterials))
                {
                    // Initiate the multipart upload request specifying the bucket and key values
                    InitiateMultipartUploadResponse initResp = cryptoClient.InitiateMultipartUpload(
                        new InitiateMultipartUploadRequest()
                        {
                            BucketName = bucketName,
                            Key = keyName
                        });

                    uploadId = initResp.UploadId;

                    long fileLength = new FileInfo(uploadSourcePath).Length;
                    long contentLength = fileLength;
                    long bytesRemaining = fileLength;


                    List<PartETag> partETags = new List<PartETag>();
                    int partNumber = 0;

                    while (bytesRemaining > 0)
                    {
                        long transferSize = bytesRemaining > partSize ? partSize : bytesRemaining;
                        long partIndex = fileLength - bytesRemaining;

                        partNumber++;

                        UploadPartResponse resp =
                            cryptoClient.UploadPart(
                                new UploadPartRequest()
                                {
                                    BucketName = bucketName,
                                    Key = keyName,
                                    FilePath = uploadSourcePath,
                                    FilePosition = partIndex,
                                    PartSize = transferSize,
                                    PartNumber = partNumber,
                                    UploadId = uploadId,
                                    IsLastPart = transferSize < AwsS3FileSystemSample1.Program.partSize
                                });

                        partETags.Add( new PartETag( partNumber, resp.ETag ));

                        bytesRemaining -= transferSize;
                    }

                    // Now complete the transfer
                    CompleteMultipartUploadResponse compResp = cryptoClient.CompleteMultipartUpload(
                        new CompleteMultipartUploadRequest()
                        {
                            Key = keyName,
                            BucketName = bucketName,
                            UploadId = initResp.UploadId,
                            PartETags = partETags
                        });
                }
            }
        }

        Console.WriteLine("Press any key to continue...");
        Console.ReadKey();
    }

Apologies for any errors and any help would be greatly appreciated.

Answer 1

After much testing and a little code spelunking on git hub in the AWS SDK for .NET source code the magic secret here is setting the PartSize member of UploadPartRequest to zero (0) when setting the IsLastPart member to true.

Exactly why this works is a matter for some debate. Since the data of the last part is typically padded by the crypto-engine to meet a cipher block boundary, the actual Content-Length is unknown until after encryption completes. Perhaps setting PartSize to zero allows the underlying code to set Content-Length to the padded length of the cipher-text rather than the value give in PartSize . Why this isn't done automatically when IsLastPart is set to true is a mystery.

In any event, the following summary may help. When using client-side encryption and multi-part uploads, set the PartSize member to zero (0) and IsLastPart to true when uploading the last part of an object's data.

This code fragment might help.

      while (bytesRemaining > 0)
      {
          long transferSize = bytesRemaining > partSize ? partSize : bytesRemaining;
          long partIndex = fileLength - bytesRemaining;

          bytesRemaining -= transferSize;
          bool isLastPart = bytesRemaining == 0;

          partNumber++;

          UploadPartResponse resp =
              cryptoClient.UploadPart(
                  new UploadPartRequest()
                  {
                      BucketName   = bucketName,
                      Key          = keyName,
                      FilePath     = uploadSourcePath,
                      FilePosition = partIndex,
                      PartSize     = isLastPart ? 0 : transferSize,
                      PartNumber   = partNumber,
                      UploadId     = uploadId,
                      IsLastPart   = isLastPart 
                  });

          partETags.Add( new PartETag( partNumber, resp.ETag ));
      }

I hope this helps someone else out there.

Answer 2

Thank you for trying to use to my implementation in the linked post.

Although I haven't read anything to explicitly support this, I think the Amazon S3 Encryption Client might be incompatible with Multipart Uploads. Although it's not definitive, I haven't been able to find a Java example (which the SDK has a KMS implementation for) using both the Amazon S3 Encryption Client on Multipart Uploads. The reason I have my doubts is that the file is encrypted using chain blocking and if each part of a Multipart Upload is encrypted piecemeal, the chain blocking would be broken and the individual initialization vectors for each piece would be lost for decryption.

I guess the only way to know is to test it out with Java first to be sure.