从c＃中的HTTPWebResponse.GetResponseStream（）上传到S3

Question

I am trying to upload from an HTTP stream directly to S3, without storing in memory or as a file first. I am already doing this with Rackspace Cloud Files as HTTP to HTTP, however the AWS authentication is beyond me so am trying to use the SDK.

The problem is the upload stream is failing with this exception:

"This stream does not support seek operations."

I've tried with PutObject and TransferUtility.Upload , both fail with the same thing.

Is there any way to stream into S3 as the stream comes in, rather than buffering the whole thing to a MemoryStream or FileStream ?

or is there any good examples of doing the authentication into S3 request using HTTPWebRequest, so I can duplicate what I do with Cloud Files?

Edit: or is there a helper function in the AWSSDK for generating the authorization header?

CODE:

This is the failing S3 part (both methods included for completeness):

string uri = RSConnection.StorageUrl + "/" + container + "/" + file.SelectSingleNode("name").InnerText;
var req = (HttpWebRequest)WebRequest.Create(uri);
req.Headers.Add("X-Auth-Token", RSConnection.AuthToken);
req.Method = "GET";

using (var resp = req.GetResponse() as HttpWebResponse)
{
    using (Stream stream = resp.GetResponseStream())
    {
        Amazon.S3.Transfer.TransferUtility trans = new Amazon.S3.Transfer.TransferUtility(S3Client);
        trans.Upload(stream, config.Element("root").Element("S3BackupBucket").Value, container + file.SelectSingleNode("name").InnerText);

        //Use EITHER the above OR the below

        PutObjectRequest putReq = new PutObjectRequest();
        putReq.WithBucketName(config.Element("root").Element("S3BackupBucket").Value);
        putReq.WithKey(container + file.SelectSingleNode("name").InnerText);
        putReq.WithInputStream(Amazon.S3.Util.AmazonS3Util.MakeStreamSeekable(stream));
        putReq.WithMetaData("content-length", file.SelectSingleNode("bytes").InnerText);

        using (S3Response putResp = S3Client.PutObject(putReq))
        {

        }
    }

}

And this is how I do it successfully from S3 to Cloud Files:

using (GetObjectResponse getResponse = S3Client.GetObject(new GetObjectRequest().WithBucketName(bucket.BucketName).WithKey(file.Key)))
{
    using (Stream s = getResponse.ResponseStream)
    {
        //We can stream right from s3 to CF, no need to store in memory or filesystem.                                            
        var req = (HttpWebRequest)WebRequest.Create(uri);
        req.Headers.Add("X-Auth-Token", RSConnection.AuthToken);
        req.Method = "PUT";

        req.AllowWriteStreamBuffering = false;
        if (req.ContentLength == -1L)
            req.SendChunked = true;


        using (Stream stream = req.GetRequestStream())
        {
            byte[] data = new byte[32768];
            int bytesRead = 0;
            while ((bytesRead = s.Read(data, 0, data.Length)) > 0)
            {
                stream.Write(data, 0, bytesRead);
            }
            stream.Flush();
            stream.Close();
        }
        req.GetResponse().Close();
    }
}   

Answer 1

As no-one answering seems to have done it, I spent the time working it out based on guidance from Steve's answer:

In answer to this question "is there any good examples of doing the authentication into S3 request using HTTPWebRequest, so I can duplicate what I do with Cloud Files?", here is how to generate the auth header manually:

string today = String.Format("{0:ddd,' 'dd' 'MMM' 'yyyy' 'HH':'mm':'ss' 'zz00}", DateTime.Now);

string stringToSign = "PUT\n" +
    "\n" +
    file.SelectSingleNode("content_type").InnerText + "\n" +
    "\n" +
    "x-amz-date:" + today + "\n" +
    "/" + strBucketName + "/" + strKey;

Encoding ae = new UTF8Encoding();
HMACSHA1 signature = new HMACSHA1(ae.GetBytes(AWSSecret));
string encodedCanonical = Convert.ToBase64String(signature.ComputeHash(ae.GetBytes(stringToSign)));

string authHeader = "AWS " + AWSKey + ":" + encodedCanonical;

string uriS3 = "https://" + strBucketName + ".s3.amazonaws.com/" + strKey;
var reqS3 = (HttpWebRequest)WebRequest.Create(uriS3);
reqS3.Headers.Add("Authorization", authHeader);
reqS3.Headers.Add("x-amz-date", today);
reqS3.ContentType = file.SelectSingleNode("content_type").InnerText;
reqS3.ContentLength = Convert.ToInt32(file.SelectSingleNode("bytes").InnerText);
reqS3.Method = "PUT";

Note the added x-amz-date header as HTTPWebRequest sends the date in a different format to what AWS is expecting.

From there it was just a case of repeating what I was already doing.

Answer 2

Take a look at Amazon S3 Authentication Tool for Curl . From that web page:

Curl is a popular command-line tool for interacting with HTTP services. This Perl script calculates the proper signature, then calls Curl with the appropriate arguments.

You could probably adapt it or its output for your use.

Answer 3

I think the problem is that according to the AWS Documentation Content-Length is required and you don't know what the length is until the stream has finished.

(I would guess the Amazon.S3.Util.AmazonS3Util.MakeStreamSeekable routine is reading the whole stream into memory to get around this problem which makes it unsuitable for your scenario.)

What you can do is read the file in chunks and upload them using MultiPart upload .

PS, I assume you know the C# source for the AWSSDK for dotnet is on Github .

Answer 4

This is a true hack (which would probably break with a new implementation of the AWSSDK), and it requires knowledge of the length of the file being requested, but if you wrap the response stream as shown with this class (a gist) as shown below:

long length = fileLength;  

you can get file length in several ways. I am uploading from a dropbox link, so they give me the length along with the url. Alternatively, you can perform a HEAD request and get the Content-Length.

string uri = RSConnection.StorageUrl + "/" + container + "/" + file.SelectSingleNode("name").InnerText;
var req = (HttpWebRequest)WebRequest.Create(uri);
req.Headers.Add("X-Auth-Token", RSConnection.AuthToken);
req.Method = "GET";

using (var resp = req.GetResponse() as HttpWebResponse)
{
    using (Stream stream = resp.GetResponseStream())
    {
        //I haven't tested this path
        Amazon.S3.Transfer.TransferUtility trans = new Amazon.S3.Transfer.TransferUtility(S3Client);
        trans.Upload(new HttpResponseStream(stream, length), config.Element("root").Element("S3BackupBucket").Value, container + file.SelectSingleNode("name").InnerText);

        //Use EITHER the above OR the below
        //I have tested this with dropbox data
        PutObjectRequest putReq = new PutObjectRequest();
        putReq.WithBucketName(config.Element("root").Element("S3BackupBucket").Value);
        putReq.WithKey(container + file.SelectSingleNode("name").InnerText);
        putReq.WithInputStream(new HttpResponseStream(stream, length)));
        //These are necessary for really large files to work
        putReq.WithTimeout(System.Threading.Timeout.Infinite);
        putReq.WithReadWriteTimeout(System.Thread.Timeout.Infinite);


        using (S3Response putResp = S3Client.PutObject(putReq))
        {

        }
    }

}

The hack is overriding the Position and Length properties, and returning 0 for Position{get}, noop'ing Position{set}, and returning the known length for Length.

I recognize that this might not work if you don't have the length or if the server providing the source does not support HEAD requests and Content-Length headers. I also realize it might not work if the reported Content-Length or the supplied length doesn't match the actual length of the file.

In my test, I also supply the Content-Type to the PutObjectRequest, but I don't that that is necessary.

Answer 5

As sgmoore said, the problem is that your content length is not seekable from the HTTP response. However HttpWebResponse does have a content length property available. So you can actually form your Http post request to S3 yourself instead of using the Amazon library.

Here's another Stackoverflow question that managed to do that with what looks like full code to me.