Sent large base64string into jsonbody

Question

I doing some integration using elastic search. I need to convert a very large file into base64string. Once its converted, then I need to sent it to ES. Below is my code.

    public IRestResponse ESMapFileAttachment(string indexName, string mappingName, string fileIndex, string fileName, Byte[] fileBytes)
    {
        var client = new RestClient(string.Format("{0}{1}/{2}/{3}", ConfigurationManager.AppSettings["esserver"], indexName, mappingName, fileIndex));
        var request = new RestRequest(Method.POST);
        request.AddHeader("Content-Type", "application/json");

        string encodedFileContent = Convert.ToBase64String(fileBytes); // got out of memory exception here
        var jsonString = @"
                                {
                                    ""file"": {
                                        ""_content"": ""XEncodedFileContentX"",
                                        ""_name"": ""XFilenameX"",
                                        ""_detect_language"": true,
                                        ""_indexed_chars"": -1
                                    }
                                }
                          ".Replace("XEncodedFileContentX", encodedFileContent)
                           .Replace("XFilenameX", fileName);

        var jsSerializer = new JavaScriptSerializer();
        jsSerializer.MaxJsonLength = Int32.MaxValue;
        var jObject = jsSerializer.DeserializeObject(jsonString);
        request.AddJsonBody(jObject);

        return client.Execute(request);
    }

But somehow I got a 'Out of Memory' exception when I tried to convert it to base64string. I already tried files below 10Mb and it works just fine. But when it really large, even around 100Mb, I got the exception, even my PC is 8GB RAM.

Answer 1

Base64 is something that will take 3 bytes of your input and converts it to 4 bytes of text on output. This is good for you because it means that you can take a smaller chunk of your input data and do base64 conversion and then you can take the very next chunk of your input and do base64 conversion and when you append the second output of base64 operation to the first one, you will have the same result as if you converted the whole thing at once.

So, what will solve your problem is to take a buffer of fixed size, divisible by 3, say 3 * 1024 * 1024 bytes for example. And you will use it for conversion of 3 MB of your input data in every round. After each round you append that to StringBuilder and you will be able to create that large result you are looking for.

Having said that, I am not sure what will then your

.Replace("XEncodedFileContentX", encodedFileContent)

cause when the encodedFileContent will be of that size. You might need to find alternative way for that part too ...

Answer 2

If you need to store whole file in memory you will need to change target platform to x64. Another way is to convert file using smaller parts. Characters for UTF8 encoding can take up to 4 bytes per char. I run an example app to convert 170MB binary file, it converted file to 226.000 characters string and app consumed about 1GB RAM.