How to convert chunck by chunck byte array into base64 string?

Question

We are reading byte array of a file and converting it into base64 string as follows

    public static string ZipToBase64()
            {
                FileUpload fileCONTENT = FindControl("FileUploadControl") as FileUpload;

                byte[] byteArr = fileCONTENT.FileBytes;

                return Convert.ToBase64String(byteArr);

            }

    string attachmentBytes = ZipToBase64();
    string json1 = "{ \"fileName\": \"Ch01.pdf\", \"data\": " + "\"" + attachmentBytes + "\"}";

When we try to convert large file upto 1 GB into base64 string, it is throwing out of memory exception. We are sending this json to restful wcf service. Following is my method in RESTful WCF Service.

 [OperationContract]
        [WebInvoke(Method = "POST", BodyStyle = WebMessageBodyStyle.WrappedRequest, RequestFormat = WebMessageFormat.Json, ResponseFormat = WebMessageFormat.Json)]
        public void UploadFile1(Stream input)
        {

            string UserName = HttpContext.Current.Request.Headers["UserName"];
            string Password = Sql.ToString(HttpContext.Current.Request.Headers["Password"]);
            string sDevideID = Sql.ToString(HttpContext.Current.Request.Headers["DeviceID"]);
            string Version = string.Empty;
            if (validateUser(UserName, Password, Version, sDevideID) == Guid.Empty)
            {
                SplendidError.SystemWarning(new StackTrace(true).GetFrame(0), "Invalid username or password for " + UserName);
                throw (new Exception("Invalid username or password for " + UserName));
            }


            string sRequest = String.Empty;
            using (StreamReader stmRequest = new StreamReader(input, System.Text.Encoding.UTF8))
            {
                sRequest = stmRequest.ReadToEnd();
            }
            // http://weblogs.asp.net/hajan/archive/2010/07/23/javascriptserializer-dictionary-to-json-serialization-and-deserialization.aspx

            JavaScriptSerializer json = new JavaScriptSerializer();
            // 12/12/2014 Paul.  No reason to limit the Json result. 
            json.MaxJsonLength = int.MaxValue;

            Dictionary<string, string> dict = json.Deserialize<Dictionary<string, string>>(sRequest);
            string base64String = dict["data"];
            string fileName = dict["fileName"];


            byte[] fileBytes = Convert.FromBase64String(base64String);
            Stream stream = new MemoryStream(fileBytes);

            //FileStream fs1 = stream as FileStream;

            string networkPath = WebConfigurationManager.AppSettings["NetWorkPath"];
            File.WriteAllBytes(networkPath + "/" + fileName, fileBytes); // Requires System.IO
        }

Please provide solution for converting large byte array into base64 string

Answer 1

The fact you use Stream input in your wcf service does not really mean you pass anything in streamed manner. In fact in your case you do not, because:

1. You buffer whole file on client in memory to build json string.
2. You buffer whole file on server in memory via stmRequest.ReadToEnd() .

So no streaming takes place. First you should realize that no json is needed here - you just need to pass file in your http request body. What you should do first is throw away all code below security checks in your UploadFile1 method and instead do this:

public void UploadFile1(string fileName, Stream input) {
    // check your security headers here
    string networkPath = WebConfigurationManager.AppSettings["NetWorkPath"];
    using (var fs = File.Create(networkPath + "/" + fileName)) {
        input.CopyTo(fs);
    }
}

Here we just copy input stream to output stream (file) without any buffering (of course CopyTo will buffer chunks but they will be very small). Mark your service method with:

[WebInvoke(Method = "POST", UriTemplate = "/UploadFile1/{fileName}")]

To allow you pass filename in query string.

Now to client. Not sure which method you use to communicate with server, I'll show example with raw HttpWebRequest.

var filePath = "path to your zip file here";
var file = new FileInfo(filePath);
// pass file name in query string
var request = (HttpWebRequest)WebRequest.Create("http://YourServiceUrl/UploadFile1/" + file.Name);            
request.Method = "POST";
// set content length
request.ContentLength = file.Length;
// stream file to server
using (var fs = File.OpenRead(file.FullName)) {
    using (var body = request.GetRequestStream()) {
         fs.CopyTo(body);
    }
}
// ensure no errors
request.GetResponse().Dispose();