Amazon S3 TransferUtility.Upload hangs in C#

Question

So I'm writing a migration application, to take some data from our local storage and upload it to Amazon. Everything is working fine, except once I get into files that are greater than 15 megs (megs, yes, NOT Gigs), the application freeze.

This is in C#, pretty straightforward.

var transferRequest = new TransferUtilityUploadRequest
    {
        Key = firstKey,
        FilePath = fileName,
        BucketName = ContentBucket,
        Timeout = 3600000,
        ContentType = GetContentTypeForFileExtension(fileName)
    };

transferRequest.UploadProgressEvent += DisplayFileProgress;
transferUtil.Upload(transferRequest);

Like I said, works just fine for files 15 megs or smaller...but on larger ones, it just stops and sits on the "Upload" command forever. 15 megs takes like 40 seconds, so I expected the 30 meg test file to take, maybe 2 minutes...but 10 minutes later, no love.

Any advice would be appreciated, as unfortunately, I will be dealing with lots of files that are 50+ megs in size.

Note that if I'm in the AWS Explorer in Visual Studio .net, I can manually upload files of 50+ megs without any issue and relatively quickly.

---

So this is "interesting"...On further review, my 50 meg files are uploading just fine. Its the code I have attached to the UploadProgressEvent that is actually causing things to freeze up, because if I comment it out, then the 50 meg files upload without issue.

If I leave this code in, 15 meg files show their progress on a progress bar. But anything bigger than 15 megs actually causes the whole app to freeze up. Can anyone tell me what might be the problem with the code that handles the progress bar updating?

private void DisplayFileProgress(object sender, UploadProgressArgs args)
{
    pbFileProgress.Invoke((MethodInvoker)delegate { 
        pbFileProgress.Value = args.PercentDone; 
        pbFileProgress.Refresh(); }); 
}

And I'm just setting " transferRequest.UploadProgressEvent += DisplayFileProgress ". Like I said, what's weird is that this works fine for smaller files but locks everything up for bigger ones.

Answer 1

Try using BeginUpload method instead of Upload.

        transferUtility.BeginUpload(request, new AsyncCallback(uploadComplete), null );
    }
    private void uploadComplete(IAsyncResult result)
    {
        var x = result;
    }

Setup your transfer utility and UploadProgressEvent as you did before. Use the Invoke method within the progress handler as you did. If you use BeginUpdate() instead of Update() it will prevent the app from hanging on the first update to your form. I couldn't find this solution anywhere so I hope it works for you.

Answer 2

Is DisplayFileProgress thread-safe? I believe (looking at some older code) that the callback is called by each upload thread independently.

This code below is part of a small utility that we use to upload files ranging from 5 MB up to 1-2 GB or so. It's not remarkably different than yours, but perhaps it might help.

var writerlock = new object();

using (var tu = new TransferUtility(amazonS3Client, tuconfig))
{
    var turequest = new TransferUtilityUploadRequest()
        .WithBucketName(bucket)
        .WithFilePath(file)
        .WithKey(Path.GetFileName(file))
        .WithStorageClass(S3StorageClass.ReducedRedundancy)
        .WithPartSize(5 * 1024 * 1024)
        .WithAutoCloseStream(true)
        .WithCannedACL(S3CannedACL.PublicRead);

    tuconfig.NumberOfUploadThreads = Environment.ProcessorCount - 1;

    // show progress information if not running batch
    if (interactive)
    {
        turequest.UploadProgressEvent += (s, e) =>
                                          {
                                              lock (writerlock)
                                              {
                                                  ... our progress routine ...
                                              }
                                          };
    }

    tu.Upload(turequest);
}