Stored procedure transferring new records to linked server losing connection

Question

Let me preface this by saying I am not a sql expert. Thanks in advance for reviewing my question.

I am having some trouble with a stored procedure that periodically loses it's connection to a linked server in the middle of transferring new records to a table there (a lot of records). I'm wondering about some things.

First, is there a better way to complete the task that is faster and more efficient?

And second, is there a way to better handle the loss of connection attempting to re-try the previous step in an effort to continue where it left off, possibly after waiting 15 or so minutes in case there is some temporary disconnect with the linked server (that may last more than what is allowed before terminating)?

Also I'm curious, when something like the highlighted step takes place, if the sql is being executed against the linked server, and the connection is lost, does sql attempt to reconnect and continue for a timeout period or since this is a compare statement does it just break? I don't really understand how it works.

Here is the section where connection loss is most common, usually occurring in the first insert statement (first step within begin catch block):

UPDATE [LINKEDSERVER.XYZ.COM].dest_database1.dbo.run
  SET last_result = 32
  WHERE type = 158;

BEGIN TRY
    INSERT INTO [LINKEDSERVER.XYZ.COM].[dest_database2].dbo.table1 ( CID , BID , Question1 , Question2 , Question3 , Question4 , Question5 , Question6 , Question7 , Question8 , Question9 , Question10 , Comments , EmailAddress , Name , Address , Address2 , City , State , Zip ) 
    SELECT src.CID , src.BID , src.Question1 , src.Question2 , src.Question3 , src.Question4 , src.Question5 , src.Question6 , src.Question7 , src.Question8 , src.Question9 , src.Question10 , src.Comments , src.EmailAddress , src.Name , src.Address , src.Address2 , src.City , src.State , src.Zip 
      FROM table1 AS src LEFT OUTER JOIN [LINKEDSERVER.XYZ.COM].[dest_database2].dbo.table1 AS dst ON src.CID = dst.CID AND src.BID = dst.BID
      WHERE dst.CID IS NULL;

    INSERT INTO [LINKEDSERVER.XYZ.COM].[dest_database2].dbo.table2 ( CID , AccountNumber , Name , Address , Address2 , City , State , Zip , BShort, EmailAddress ) 
    SELECT src.CID , src.AccountNumber , src.Name , src.Address , src.Address2 , src.City , src.State , src.Zip , src.BShort, src.EmailAddress 
      FROM table2 AS src LEFT OUTER JOIN [LINKEDSERVER.XYZ.COM].[dest_database2].dbo.table2 AS dst ON src.BShort = dst.BShort
      WHERE dst.BShort IS NULL;
END TRY
BEGIN CATCH
    SELECT @ErrorCode = @@Error;
    SELECT @ErrorResult = 109;
    SELECT @ErrorMessage = 'Failed Copy ' + CAST ( @ErrorCode AS varchar ) ;
    GOTO ExitWithError;
END CATCH;

INSERT INTO [LINKEDSERVER.XYZ.COM].dest_database1.dbo.system_log ( notes , log_type , source_type , parent_id ) 
VALUES ( 'Copied tables xyz ' , 45 , 58 , 0 ) ;

UPDATE [LINKEDSERVER.XYZ.COM].dest_database1.dbo.run
  SET last_result = 31
  WHERE type = 158;

I understand that if the connection is lost, my catch block doesn't execute, so ExitWithError isn't used. And I think I understand when the connection is lost, a rollback occurs (not sure how/if this works with linked servers because some of the records are transferred when this happens), and it just quits executing everything - because neither the catch is executed or the statements after the entire begin catch block. I was hoping to, in the event this occurs, log something so that the stored procedure, when it runs again (set to run every 30 min until successful), it can pick up where it left off, because I have code in the beginning that looks for the last status/log entries and determines what to do. Is there a better way to handle this?

Answer 1

So first tip, Do the stored procedure from the other server (destination) if you can. In other words switch the local and the linked server. The reason is I have had significantly improved performance to a DML (insert/update/delete) command from a Linked Server to a local table. When doing the DML into a linked table the performance stinks!

Next effectively use the ids, are they sequential? Can you just take the MAX of what is at the destination server instead of having to Join large tables between the servers?

If you cannot count on sequential then you can build a temp table on the source server and filter there instead of trying to do it across tables. If just ids you can also try to pass XML in nvarchar format back and forth. Or you can maintain a tracking table on the source so you know what ids have already been processed using the OUTPUT inserted.id INTO .... from your insert statement.

Finally set your procedure to batch/cut up and do the transactions in smaller chunks such as SELECT TOP x# of records at a time. You can still place the whole thing in a while loop so that it does it all in one execution of the proc, but then you can do your transactions in smaller more manageable pieces.