使用 Newtonsoft.Json 将 Json 直接序列化到 AWS S3 存储桶中

Question

I have an object that has to be converted to Json format and uploaded via Stream object. This is the AWS S3 upload code:

        AWSS3Client.PutObjectAsync(new PutObjectRequest()
        {
            InputStream = stream,
            BucketName = name,
            Key = keyName
        }).Wait();

Here stream is Stream type which is read by AWSS3Client. The data that I am uploading is a complex object that has to be in Json format.

I can convert object to string using JsonConvert.SerializeObject or serialize to file using JsonSerializer but since amount of data is quite significant I would prefer to avoid temporary string or file and convert object to readable Stream right away. My ideal code would look something like this:

        AWSS3Client.PutObjectAsync(new PutObjectRequest()
        {
            InputStream = MagicJsonConverter.ToStream(myDataObject),
            BucketName = name,
            Key = keyName
        }).Wait();

Is there a way to achieve this using Newtonsoft.Json ?

Answer 1

你在这里需要两件事：一个是生产者/消费者流，例如来自这个 StackOverflow 问题的BlockingStream ，第二个是 Json.Net 序列化程序写入这个流，就像在另一个 SO 问题中一样。

Answer 2

Another practical option is to wrap the memory stream with gzip stream (2 lines of code).
Usually, JSON files will have great compression (1GB file can be compressed to 50MB).
Then when serving the stream to S3, wrap it with gzip stream which decompresses it.
I guess the trade-off comparing to temp file is CPU vs IO (both will probably work well). If you can save it compressed on S3 it will save you space and increase networking efficiency too.
Example code:

var compressed = new MemoryStream();
using (var zip = new GZipStream(compressed, CompressionLevel.Fastest, true))
{
    -> Write to zip stream...
}
compressed.Seek(0, SeekOrigin.Begin);
-> Use stream to upload to S3