Send Large Data with WCF without Streaming

Question

We are trying to build a web service that can send/receive large data files from kb~gb sizes.

I know that we should use streaming to preform that so is not going to crash the memory.

But problem is we don't want to change our design, or more like, now we are going to try to see what will happen when we transfers large file say just 150mb file in a byte[] buffered transfers mode with WCF. (I know is goign to buffer the whole file into memory... and crash/exception if we transfer gb size files...)

But even so, they want to see what happen, so I have my wsHttpBinding in WCF config:

  <system.web>
    <compilation debug="true"/>
      <httpRuntime maxRequestLength="524288"/>
  </system.web>
....

  <wsHttpBinding>
    <binding name="wsHttpBinding1" closeTimeout="00:10:00" openTimeout="00:10:00"
          receiveTimeout="00:30:00" sendTimeout="00:30:00" bypassProxyOnLocal="false"
          hostNameComparisonMode="StrongWildcard"                  
                                    maxBufferPoolSize="52428800" maxReceivedMessageSize="65536000"
          messageEncoding="Mtom" useDefaultWebProxy="true">
      <security mode="None" />
    </binding>
  </wsHttpBinding>

Then my client app config:

          <wsHttpBinding>
                <binding name="WSHttpBinding_IPrintService" closeTimeout="00:01:00"
                      openTimeout="00:01:00" receiveTimeout="00:30:00" sendTimeout="00:30:00"
                      bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard"
                                    maxBufferPoolSize="52428800" maxReceivedMessageSize="65536000"
                      messageEncoding="Mtom" textEncoding="utf-8" useDefaultWebProxy="true"                           
                      allowCookies="false">
                      <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
                            maxBytesPerRead="4096" maxNameTableCharCount="16384" />
                      <reliableSession ordered="true" inactivityTimeout="00:10:00"
                            enabled="false" />
                  <security mode="None"/>
                </binding>
          </wsHttpBinding>

When I transfers small data it works fine, but when I try to transfers 150mb file, it gives me exceptions:

InnerException = {"Unable to write data to the transport connection: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full."}

I tried to increase maxReceivedMessage and such, but it still having the same problem....

======

Edit: I tried to increase all possible value above to... 2147483647.... I can sent a 30mb file successfully..but still not 150mb.... And I also tried to use this CustomBinding:

<customBinding>         
        <binding name="LargeCustomBinding" 
                 receiveTimeout="01:00:00"  
                 openTimeout="01:00:00"  
                 closeTimeout="01:00:00"  
                 sendTimeout="01:00:00"> 
          <binaryMessageEncoding  
            maxReadPoolSize="2147483647"  
            maxSessionSize="2147483647"  
            maxWritePoolSize="2147483647"> 
            <readerQuotas  
              maxArrayLength="2147483647"  
              maxDepth="2147483647"  
              maxBytesPerRead="2147483647"  
              maxNameTableCharCount="2147483647"  
              maxStringContentLength="2147483647" /> 
          </binaryMessageEncoding> 
          <httpTransport  
            maxBufferSize="2147483647"  
            maxReceivedMessageSize="2147483647"  
            maxBufferPoolSize="2147483647"/> 
        </binding> 
      </customBinding>

But I am still getting the same problem....

=========

Edit 2: I guess I should put concern on how I sending the data, maybe I was wrong, I have an Object MyData:

    byte[] LargeData = File.ReadAllBytes(this.LargeDataPath.Text);
    MyData.LargeData = new List<KeyValuePair<string, byte[]>>()
    {
        new KeyValuePair<string, byte[]>("DATA", LargeData)
    };

    myServiceClient.SendDataAsync( new SendDataRequest(MyData));

There should be no problem? All I did is just sending one big byte[] inside a list.....

Answer 1

You should be sending a smaller chunks of data with a separate send method calls.

Put the data you want to send into a byte array. Then realize the logic on both ends, that will know how to handle sizes and just start sending the data in a smaller pieces, one by one. When you have sent an entire file, it will get concatenated into an array on a client and you will be able to do further processing.

Answer 2

Fix this first:

maxStringContentLength="8192"
maxArrayLength="16384"

And ensure you have

maxStringContentLength="2147483646"
maxArrayLength="2147483646"

On both client and server.

I was sending 30Mb over WCF without problems.

And check out this