Row by row streaming of data while waiting for response

Question

I have a WCF service and client developed using C# which handles data transfer from SQL server database on the server to the SQL server database on the client end. I am facing some issues with the current architecture and planning to modify it to an idea I have, and would like to know if it is possible to achieve it, or how best can I modify the architecture to suite my needs.

The Server side database server is SQL 2008 R2 SP1 and client side servers are SQL 2000

Before I state the idea, below is the overview and current shortcomings of the architecture design I am using.

Overview:

1. Client requests for a table's data.

2. WCF service queries the Server database for all pending data for the requested table. This data is loaded into a dataset.

3. WCF Compresses the Dataset using GZIP compression and converts it to byte for the client to download.

4. Client receives the Byte stream, un-compresses it and replicates the data from the Dataset to the physical table on the client database. This data is inserted row by row since in need the Primary key column filed to be returned to the server so that it can be flagged of as transferred.

5. Once the client has finished replicating the data, it uploads the successful rows Primary key fields back to the server, and in turn the server update each field one by one. The above procedure uses a basic http binding, with streamed transfer mode.

Shortcomings:

1. This works great for little data, but when it comes to bulk data, maintaining the dataset in memory as the download is ongoing and also at the client side as replication is ongoing, is becoming impossible as sometimes the dataset size goes up to 4gb. The server can hold this much data since it's a 32gb RAM server, but at the client side I get System out of memory exception since the client machine has 2gb RAM.

2. There are numerous deadlocks as the select query is running and also when updating since I am using transaction mode as read committed.

3. For bulk data it is very slow and completely hangs the client machine when the DTS is ongoing.

Idea in mind:

Maintain the same service and logic of row by row transfer since I cannot change this due to sensitivity of the data, but rather than downloading bulk data I plan to use the sample given in http://code.msdn.microsoft.com/Custom-WCF-Streaming-436861e6 . Thus the new flow will be as:

1. Upon receiving the download request, the server will open a connection to the DB using snapshot isolation as the transaction level.

2. Build the row by row object on the server and send it to the client on the requested channel, as the client receives each row object, it gets processed and a success or failure response is sent back to the server on the same method same channel, as I need to update the data on the same snapshot transaction. This way I will reduce bulk objects in memory, and rely on SQL for the snapshot data that will be maintained in temdb once the transaction is initiated.

Challenge:

1. How can I send the row object and wait for a confirmation before sending the next one, as the update to the server row has to occur on the same snapshot transaction. Since if I create another method on the service to perform the flagging off, the snapshots will be different and this will cause issues in the integrity of the data in case the data undergoes changes after the snapshot transaction was initiated.

2. If this is the wrong approach, then please suggest a better one, as I am open to any suggestions.

3. If my understanding of the snapshot isolation is wrong, then please correct me as I am new to this.

Update 1:

I would like to achieve something like this when the client is the one requesting:

    //Client Invokes this method on the server
    public Stream GetData(string sTableName)
    {
        //Open the Snapshot Transaction on the Server
        SqlDataReader rdr = operations.InitSnapshotTrans("Select * from " + sTableName + " Where Isnull(ColToCheck,'N') <> 'Y'");

        //Check if there are rows available
        if(rdr.HasRows)
        {
            while rdr.read()
            {
                SendObj sendobj = Logic.CreateObejct(rdr);
                //Here is where i am stuck
                //At this point I want to write the object to the Stream
                ...Write sendobj to Stream
                //Once the client is done processing it reverts with a true for success or false for failuer.
                if (returnObj == true)
                {
                    operations.updateMovedRecord(rdr);
                }
            }
        }
    }

For the server sending i have written the code as Such (I used Pub Sub Model for this):

    public void ServerData(string sServerText)
    {
        List<SubList> subscribers = Filter.GetClients();
        if (subscribers == null) return;

        Type type = typeof(ITransfer);
        MethodInfo publishMethodInfo = type.GetMethod("ServerData");

        foreach (SubList subscriber in subscribers)
        {
            try
            {
                //Open the Snapshot Transaction on the Server
                SqlDataReader rdr = operations.InitSnapshotTrans("Select * from " + sTableName + " Where Isnull(ColToCheck,'N') <> 'Y'");

                //Check if there are rows available
                if(rdr.HasRows)
                {
                    while rdr.read()
                    {
                        SendObj sendobj = Logic.CreateObejct(rdr);

                        bool rtnVal = Convert.ToBoolean(publishMethodInfo.Invoke(subscriber.CallBackId, new object[] { sendobj }));

                        if (rtnVal == true)
                        {
                            operations.updateMovedRecord(rdr);
                        }
                    }
                }               
            }
            catch (Exception ex)
            {
                Debug.WriteLine(ex.Message);
            }
        }
    }

Answer 1

Just off the top of my head, this sounds like it might take longer. That may or may not be a concern.

Given the requirement in challenge 1 (that everything happen in the context of one method call), it sounds like what actually needs to happen is for the server to call a method on the client , sending a record, and then waiting for the client to return confirmation. That way, everything that needs to happen, happens in the context of a single call (server to client). I don't know if that's feasible in your situation.

Another option might be to use some kind of double-queue system (perhaps with MSMQ?) so that the server and client can maintain an ongoing conversation within a single session.

I assume there's a reason why you can't just divide the data to be downloaded into manageable chunks and repeatedly execute the original process on the chunks. That sounds the least ambitious option, but you probably would have done it already if it met all your needs.