C# HttpWebRequest POST data in chunk

Question

What is the max length of byte array ? I'm trying to create an array of bytes whose length is 551858003 . I have created on zip file of near about 526 MB. But it gives me error Out of memory exception I am uploading the file on google drive .

Here I tried some code. I am reading the bytes of zip file by using following code.

 byte[] FileByteArray = null;

                using (IsolatedStorageFile storage = IsolatedStorageFile.GetUserStoreForApplication())
                {
                    if (storage.FileExists(fileName))
                    { 
                        using (IsolatedStorageFileStream fileStream = storage.OpenFile(fileName,FileMode.Open,FileAccess.Read))
                        {
                            fileStream.Flush();
                            fileStream.Position = 0;
                            long len = fileStream.Length;
                            FileByteArray = new byte[len];
                            fileStream.Read(FileByteArray, 0, FileByteArray.Length);

                            //using (BinaryReader binReader = new BinaryReader(fileStream))
                            //{
                            //    Int32 Filelength = Convert.ToInt32(fileStream.Length);
                            //    FileByteArray = binReader.ReadBytes(Filelength);
                            //}
                            fileStream.Flush();
                            fileStream.Dispose();
                            fileStream.Close();
                        }
                    }
                }

How can I resolve this issue ? I am getting OutOfMemoryException while uploading large file. I can upload near about 100MB.

Here is my Method for sending images in chunk

 public void Images_ChunkRequst(string uploadURL, byte[] FileByteArray, int startIndex)
    {
        try
        {
            int chunkSize = 256 * 1024 * 2;
            int totalChunks = (int)Math.Ceiling((double)FileByteArray.Length / chunkSize);
            int endIndex = (int)(startIndex + chunkSize > FileByteArray.Length ? FileByteArray.Length : startIndex + chunkSize);
            int length = endIndex - startIndex;

            if (i < totalChunks)
            {
                 CollectionIP = CheckInternet.Find();
                if (CollectionIP.Count == 2 && DeviceNetworkInformation.IsWiFiEnabled)
                    NetworkIPaddress = IPAddress.Parse(CollectionIP[1]).ToString();
                else if (CollectionIP.Count > 0 && DeviceNetworkInformation.IsWiFiEnabled)
                    NetworkIPaddress = IPAddress.Parse(CollectionIP[0]).ToString();

                if (!string.IsNullOrEmpty(NetworkIPaddress))
                {
                    i = i + 1;
                    var request = WebRequest.Create(uploadURL) as HttpWebRequest;
                    request.Method = "PUT";
                    request.Headers["Authorization"] = string.Format("Bearer {0} ", AccessToken);
                    request.ContentType = "application/zip";
                    request.ContentLength = length;
                    request.Headers["Content-Range"] = "bytes " + startIndex + "-" + (endIndex - 1) + "/" + FileByteArray.Length;                        
                    request.AllowWriteStreamBuffering = false;

                    request.BeginGetRequestStream(arPut =>
                    {
                        var request1 = (HttpWebRequest)arPut.AsyncState;
                        using (var dataStream = request1.EndGetRequestStream(arPut))
                        {
                            //getting exception here
                            dataStream.Write(FileByteArray.Skip(startIndex).Take(endIndex).ToArray(), 0, FileByteArray.Take(length).Count());                                
                            dataStream.Flush();
                            dataStream.Dispose();
                            dataStream.Close();
                        }
                        request1.BeginGetResponse(aPut =>
                        {
                            var request2 = (HttpWebRequest)aPut.AsyncState;
                            WebRequest.RegisterPrefix("http://", WebRequestCreator.ClientHttp);
                            WebRequest.RegisterPrefix("https://", WebRequestCreator.ClientHttp);
                            var response = (HttpWebResponse)request2.EndGetResponse(aPut);

                            if (response.StatusCode.ToString() == "308") // Resume Incomplete
                            {
                                response.Dispose();
                                response.Close();

                                string packet = response.Headers["Range"];
                                string[] rng = packet.Remove(0, 6).Split('-');
                                Images_ChunkRequst(uploadURL, FileByteArray, Convert.ToInt32(rng[1]) + 1);
                            }
                            else if (response.StatusCode.ToString() == "500") //Internal Server Error
                            {
                                i = i - 1;
                                response.Dispose();
                                response.Close();
                                Images_ChunkRequst(uploadURL, FileByteArray, startIndex);
                            }
                            else if (response.StatusCode.ToString() == "502") //Bad Gateway
                            {
                                i = i - 1;
                                response.Dispose();
                                response.Close();
                                Images_ChunkRequst(uploadURL, FileByteArray, startIndex);
                            }
                            else if (response.StatusCode.ToString() == "503") //Service Unavailable
                            {
                                i = i - 1;
                                response.Dispose();
                                response.Close();
                                Images_ChunkRequst(uploadURL, FileByteArray, startIndex);
                            }
                            else if (response.StatusCode.ToString() == "504") //Gateway Timeout
                            {
                                i = i - 1;

                                response.Dispose();
                                response.Close();
                                Images_ChunkRequst(uploadURL, FileByteArray, startIndex);
                            }
                            else if (response.StatusCode == HttpStatusCode.NotFound) //Not Found
                            {
                                i = i - 1;

                                response.Dispose();
                                response.Close();
                                Images_ChunkRequst(uploadURL, FileByteArray, startIndex);
                            }
                            else if (response.StatusCode == HttpStatusCode.OK) // upload complete.
                            {
                                Dispatcher.BeginInvoke(() =>
                                {
                                    i = 0;// i must be 0 after each upload success
                                    imgPass_Images.Visibility = Visibility.Visible;
                                    this.LayoutRoot.IsHitTestVisible = true;
                                    SystemTray.ProgressIndicator.IsIndeterminate = false;
                                    SystemTray.IsVisible = false;
                                    GetNextAutoOrder();
                                });
                            }
                        }, request1);
                    }, request);
                }
                else
                {
                    this.LayoutRoot.IsHitTestVisible = true;
                    SystemTray.ProgressIndicator.IsIndeterminate = false;
                    SystemTray.IsVisible = false;
                    MessageBox.Show("Please check your internet connection.");
                }
            }
        }
        catch (Exception ex)
        {
            MessageBox.Show("Error occured. Trying to send failed package.");
            Images_ChunkRequst(uploadURL, FileByteArray, startIndex);
        }
    }

I am getting exception here dataStream.Write(FileByteArray.Skip(startIndex).Take(endIndex).ToArray(), 0, FileByteArray.Take(length).Count()); . I have tried lot to solve this issue. Can some one please help me? I am getting this exception Insufficient memory to continue the execution of the program

Answer 1

The current implementation of System.Array uses Int32 for all its internal counters etc, so the theoretical maximum number of elements is Int32.MaxValue . So the problem is somewhere else.

Answer 2

This has nothing to do with Array.Length . You're running out of process memory. You should do allocations in chunks.

Answer 3

This might not be a solution , but I have some issues with the code you posted:

                    using (IsolatedStorageFileStream fileStream = storage.OpenFile(fileName,FileMode.Open,FileAccess.Read))
                    {
                        fileStream.Flush();
                        fileStream.Position = 0;
                        long len = fileStream.Length;
                        FileByteArray = new byte[len];
                        fileStream.Read(FileByteArray, 0, FileByteArray.Length);

                        //using (BinaryReader binReader = new BinaryReader(fileStream))
                        //{
                        //    Int32 Filelength = Convert.ToInt32(fileStream.Length);
                        //    FileByteArray = binReader.ReadBytes(Filelength);
                        //}
                        fileStream.Flush();
                        fileStream.Dispose();
                        fileStream.Close();
                    }

• fileStream.Flush() if forcing to write the buffer to disk. Not required for Read
• fileStream.Close() would fail because Dispose() is called before it
• fileStream.Dispose() and fileStream.Close() aren't required since it should automatic be handled by the using() statement
• fileStream.Read(array, start, lenght) returns an integer of how many bytes that was actual read, so it doesn't always match the actual length. You should keep reading until zero is returned. Which is why I suggest following:

Instead of create FileByteArray to the size of the stream, do it in chucks as well and maybe write the result to maybe a memory stream (or even better send it directly to the API if it support sending in chucks as well.) So basically:

Create a byte array of like 1024 (could be bigger if you want bigger reads from the buffer - but avoid too big)

Do the read in a loop:

byte[] buffer = new byte[1024];
int read = 0;
while((read = fileStream.Read(buffer, 0, buffer.Length)) > 0)
     // write buffer[0:read] to memory location or temp file on disk (avoid Flushing for each write);