Synchronizing table with 2mb entry with MS Sync Framework and IIS-Hosted WCF Service takes 10 minutes

Question

I have a IIS-Hosted WCF service running on one server and a windows service running on another. The windows service acts as the slave and synchronizes the databases on each of the servers by connecting to the IIS-Hosted WCF service, with the use of MS Sync Framework.

I am storing the content of files in one of the tables that is beeing synchronized, and the entries can become a bit large (~10mb per entry). My problem is that when I try to synchronize these entries it takes a long time to synchronize.

The table being synchronized:

[Id] [int] IDENTITY(1,1) NOT NULL,
[Client] [uniqueidentifier] NOT NULL,
[Date] [datetime] NOT NULL,
[Content] [nvarchar](max) NOT NULL,

I've timed the synchronization with 1 entry with different content sizes to the following;

• 500kb - 1min 20sec
• 1mb - 3min
• 1,5mb - 6min 30sec
• 2mb - 9min 50sec

(The timings were approximately the same when debugging in Visual Studio with both services and both databases locally)

Does anyone have an explanation to why this takes so long?

WCF service config:

<configuration>
  <system.web>
    <httpRuntime maxRequestLength="2147483647" />
    <compilation debug="true" targetFramework="4.0"/>
  </system.web>

  <system.serviceModel>
    <bindings>
      <basicHttpBinding>
        <binding closeTimeout="00:20:00" openTimeout="00:20:00" receiveTimeout="00:20:00" sendTimeout="00:20:00"
        allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
        maxBufferSize="2147483647" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647"
        messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
        useDefaultWebProxy="true">
          <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647"/>
          <security mode="None">
            <transport clientCredentialType="None" proxyCredentialType="None"
            realm="" />
            <message clientCredentialType="UserName" algorithmSuite="Default" />
          </security>
        </binding>
      </basicHttpBinding>
    </bindings>

    <behaviors>
      <serviceBehaviors>
        <behavior>
          <serviceMetadata httpGetEnabled="true"/>
          <serviceDebug includeExceptionDetailInFaults="true" httpHelpPageEnabled="true"/>
        </behavior>
      </serviceBehaviors>
    </behaviors>

    <serviceHostingEnvironment multipleSiteBindingsEnabled="true"/>
  </system.serviceModel>

  <system.webServer>
    <modules runAllManagedModulesForAllRequests="true"/>
  </system.webServer>

</configuration>

Windows service config:

<configuration>

  <system.serviceModel>
    <bindings>
      <basicHttpBinding>
        <binding name="BasicHttpBinding_ISyncServiceContract" closeTimeout="00:20:00" openTimeout="00:20:00" receiveTimeout="00:20:00" sendTimeout="00:20:00"
             allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
             maxBufferSize="2147483647" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647"
             messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered" useDefaultWebProxy="true">
          <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647"/>
          <security mode="None">
            <transport clientCredentialType="None" proxyCredentialType="None" realm=""/>
            <message clientCredentialType="UserName" algorithmSuite="Default"/>
          </security>
        </binding>
      </basicHttpBinding>
    </bindings>

    <client>
      <endpoint address="http://localhost:60000/SyncService.svc" binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_ISyncServiceContract" contract="SyncServiceReference.ISyncServiceContract" name="BasicHttpBinding_ISyncServiceContract"/>
    </client>
  </system.serviceModel>

  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>
  </startup>

</configuration>

Answer 1

It seems the problem is the nvarchar(max) field when it contains too much data. I changed the field to varbinary(max) and now the synchronization of the same data is over in a fraction of the time used before.

Answer 2

If you are using the client/server libraries, the communication ends up being essentially a dataset that is serialized into XML (it isn't technically "serialized", but it ends up being roughly similar). So you start with the dataset which has a reputation for being bloated. Additionally the dataset will store before and after data for each row that is changed - doubling the amount of data. Then add serialization to XML which adds more overhead. Then that payload is wrapped in web service calls - which add a little more overhead. All told, you are looking at transferring a very large amount of data across the wire. The time lapse you are seeing is just the amount of time it is taking to transfer all that data.

This goes beyond your question, but some recommendations to resolve this are:

• Look at compressing the data as it is transferred across the wire. There are some wcf + gzip projects out there.
• You may want to consider moving the files out of the database into the filesystem and using a separate WCF service to handle transferring files.