Deadlocks after extracting part of the trigger into a stored procedure

Question

Today we had very interesting case that mystifies me a lot. In a nutshell, we did some refactoring, cleaning up triggers from the repeated code, extracting it into the single and reusable stored procedure. We thought this refactoring will have no side-effects, but we were badly wrong. After the release, we encountered a lot of deadlocks and performance degradation with no evident reasons. After inspection of system tables to see what DB is taken by, we figured out that the refactoring above was involved and we ended up with rolling back the update.

We did not reproduce the issue in test environments to probe the cure yet, so some tricky conditions take place in order for issue to be visible.

The below are the details of what the change was about. We updated a lot of triggers, but all of them are very similar, I'll show you the one. It should be enough as I've found deadlock graph that shows that there is a deadlock where two processes were executing the single trigger (show below) and deadlocked.

Let me start with solution that worked before (that we rolled back to and that looks almost identical to deadlocking solution shown below as well).

CREATE TRIGGER [dbo].[TR__xyz__update_sync_publishers]
ON [dbo].[xyz]
AFTER INSERT, DELETE, UPDATE
AS 
BEGIN
    SET NOCOUNT ON;

    if(TRIGGER_NESTLEVEL() = 1)
    BEGIN
        create table #AffectedIDs (advisor_id int primary key)

        insert into #AffectedIDs
            select distinct t.id
            from
                (select id 
                 from inserted
                 inner join xyz a ON a.id = id
                 where [text] <> ''
                 union 
                 select id 
                 from inserted
                 inner join xyz a ON a.id = id
                 where [text] <> '') t

        declare @date datetime = getutcdate()
        declare @RegisteredObjectTypeID int = 2
        declare @SyncPublisherSourceID int = 1

        update pub
        set pub.master_update_date = @date
        from #AffectedIDs affected
        inner join sync_publishers pub on 
            pub.sync_registered_object_type_id = @RegisteredObjectTypeID 
            and pub.sync_publisher_source_id = @SyncPublisherSourceID
            and pub.sync_object_id = affected.advisor_id

        insert into sync_publishers (sync_object_id, sync_registered_object_type_id, sync_publisher_source_id , master_update_date)
        select 
            affected.advisor_id,
            @RegisteredObjectTypeID,
            @SyncPublisherSourceID,
            @date
        from #AffectedIDs affected
        left join sync_publishers pub on 
            pub.sync_registered_object_type_id = @RegisteredObjectTypeID
            and pub.sync_publisher_source_id = @SyncPublisherSourceID
            and pub.sync_object_id = affected.advisor_id
        where 
            pub.sync_object_id is null

        drop table #AffectedIDs
    END
END

Here is the new trigger that deadlocks.

CREATE TRIGGER [dbo].[TR__xyz__update_sync_publishers]
   ON [dbo].[xyz]
   AFTER INSERT,DELETE,UPDATE
AS 
BEGIN

    SET NOCOUNT ON;

    declare @ids dtInt

    insert into @ids
    select distinct t.id
    from
    (
        select id from inserted
        INNER JOIN xyz a ON a.id = id
        WHERE [text] <> ''
        union 
        select id from inserted
        INNER JOIN xyz a ON a.id = id
        WHERE [text] <> ''
    ) t

    exec SyncTracker_PublishEvent 2, @ids

END

Here is the definition of extracted SP:

CREATE PROCEDURE [dbo].[SyncTracker_PublishEvent]
    @objectTypeId int, 
    @ids dtInt readonly
AS
BEGIN
    SET NOCOUNT ON;

    if(TRIGGER_NESTLEVEL() > 1) RETURN;

    declare @pubSourceId int = 1
    declare @date datetime = getutcdate()

    update pub
    set pub.master_update_date = @date
    from @ids affected
    inner join sync_publishers pub
    on pub.sync_registered_object_type_id = @objectTypeId 
        and pub.sync_publisher_source_id = @pubSourceId
        and pub.sync_object_id = affected.value

    insert into sync_publishers (sync_object_id, sync_registered_object_type_id, sync_publisher_source_id , master_update_date)
    select affected.value, @objectTypeId, @pubSourceId, @date
    from @ids affected
    left join sync_publishers pub
    on pub.sync_registered_object_type_id = @objectTypeId
        and pub.sync_publisher_source_id = @pubSourceId
        and pub.sync_object_id = affected.value
    where 
        pub.sync_object_id is null
END
GO

The definition of dtInt.

CREATE TYPE [dbo].[dtInt] AS TABLE
(
    [value] [int] NOT NULL,
    PRIMARY KEY CLUSTERED 
    (
        [value] ASC
    )
)

And finally the deadlock graph.

<deadlock>
  <victim-list>
    <victimProcess id="processe1892fe8c8" />
  </victim-list>
  <process-list>
    <process id="processe1892fe8c8" taskpriority="0" logused="3824" waitresource="KEY: 5:72057602924150784 (4776e78e2961)" waittime="5686" ownerId="2583257965" transactionname="user_transaction" lasttranstarted="2016-10-03T08:30:42.500" XDES="0xe192b24408" lockMode="U" schedulerid="6" kpid="41296" status="suspended" spid="141" sbid="0" ecid="0" priority="0" trancount="2" lastbatchstarted="2016-10-03T08:30:42.503" lastbatchcompleted="2016-10-03T08:30:42.493" lastattention="2016-10-03T08:29:01.693" clientapp="..." hostname="..." hostpid="22572" loginname="kbuser" isolationlevel="read committed (2)" xactid="2583257965" currentdb="5" lockTimeout="4294967295" clientoption1="673316896" clientoption2="128056">
      <executionStack>
        <frame procname="63c1b4d8-1c55-4429-b057-81fb6da8f780.dbo.SyncTracker_PublishEvent" line="21" stmtstart="1178" stmtend="1680" sqlhandle="0x030005007bf23c4b5012b40092a6000001000000000000000000000000000000000000000000000000000000">
update pub
    set pub.master_update_date = @date
    from @ids affected
    inner join sync_publishers pub
    on pub.sync_registered_object_type_id = @objectTypeId 
        and pub.sync_publisher_source_id = @pubSourceId
        and pub.sync_object_id = affected.valu    </frame>
        <frame procname="63c1b4d8-1c55-4429-b057-81fb6da8f780.dbo.TR__xyz__update_sync_publishers" line="28" stmtstart="1300" stmtend="1372" sqlhandle="0x03000500f711233ddee4c60090a6000000000000000000000000000000000000000000000000000000000000">
exec SyncTracker_PublishEvent 2, @id    </frame>
        <frame procname="unknown" line="1" stmtstart="1054" stmtend="3032" sqlhandle="0x02000000912653235c5ef3529289f19ae4445e62ee1ccbc00000000000000000000000000000000000000000">
unknown    </frame>
        <frame procname="unknown" line="1" sqlhandle="0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000">
unknown    </frame>
      </executionStack>
    </process>
    <process id="processdfa401b848" taskpriority="0" logused="9384" waitresource="KEY: 5:72057602924150784 (1501093f83b4)" waittime="5814" ownerId="2582414029" transactionname="user_transaction" lasttranstarted="2016-10-03T08:30:09.933" XDES="0x104486ac408" lockMode="U" schedulerid="1" kpid="19548" status="suspended" spid="213" sbid="0" ecid="0" priority="0" trancount="2" lastbatchstarted="2016-10-03T08:30:53.047" lastbatchcompleted="2016-10-03T08:30:53.047" lastattention="1900-01-01T00:00:00.047" clientapp="..." hostname="..." hostpid="6196" loginname="kbuser" isolationlevel="read committed (2)" xactid="2582414029" currentdb="5" lockTimeout="4294967295" clientoption1="673316896" clientoption2="128056">
      <executionStack>
        <frame procname="63c1b4d8-1c55-4429-b057-81fb6da8f780.dbo.SyncTracker_PublishEvent" line="21" stmtstart="1178" stmtend="1680" sqlhandle="0x030005007bf23c4b5012b40092a6000001000000000000000000000000000000000000000000000000000000">
update pub
    set pub.master_update_date = @date
    from @ids affected
    inner join sync_publishers pub
    on pub.sync_registered_object_type_id = @objectTypeId 
        and pub.sync_publisher_source_id = @pubSourceId
        and pub.sync_object_id = affected.valu    </frame>
        <frame procname="63c1b4d8-1c55-4429-b057-81fb6da8f780.dbo.TR__xyz__update_sync_publishers" line="28" stmtstart="1300" stmtend="1372" sqlhandle="0x03000500f711233ddee4c60090a6000000000000000000000000000000000000000000000000000000000000">
exec SyncTracker_PublishEvent 2, @id    </frame>
        <frame procname="unknown" line="1" stmtstart="1120" stmtend="3132" sqlhandle="0x020000007414d821ed68a2ab4462b4eca6b2fdb4ba28cc350000000000000000000000000000000000000000">
unknown    </frame>
        <frame procname="unknown" line="1" sqlhandle="0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000">
unknown    </frame>
      </executionStack>
    </process>
  </process-list>
  <resource-list>
    <keylock hobtid="72057602924150784" dbid="5" objectname="63c1b4d8-1c55-4429-b057-81fb6da8f780.dbo.sync_publishers" indexname="IX__sync_publishers__registered_object_type_id__sync_object_id" id="lock10887a96b00" mode="X" associatedObjectId="72057602924150784">
      <owner-list>
        <owner id="processdfa401b848" mode="X" />
      </owner-list>
      <waiter-list>
        <waiter id="processe1892fe8c8" mode="U" requestType="wait" />
      </waiter-list>
    </keylock>
    <keylock hobtid="72057602924150784" dbid="5" objectname="63c1b4d8-1c55-4429-b057-81fb6da8f780.dbo.sync_publishers" indexname="IX__sync_publishers__registered_object_type_id__sync_object_id" id="lockdb7d7b8200" mode="X" associatedObjectId="72057602924150784">
      <owner-list>
        <owner id="processe1892fe8c8" mode="X" />
      </owner-list>
      <waiter-list>
        <waiter id="processdfa401b848" mode="U" requestType="wait" />
      </waiter-list>
    </keylock>
  </resource-list>
</deadlock>

The definition of sync_publishers available here: http://pastebin.com/LviwwCDi .

If you have any thoughts on feasible causes - please welcome to share - we will highly appreciate that!

UPDATE 1. Actual execution plans for UPDATE/INSERT into sync_publishers

Actual execution plans looks pretty much identical.

New exec plan (that occasionally deadlocks).

Old exec plan (that does not).

UPDATE 2. Tried some advises

I did try some advises today:

Got rid of "key lookups" in query plans due to missing sync_publisher_source_id within the non-clustered index by deletion column entirely -- it was not really mandatory in our implementation.

Rewrote UPDATE + INSERT as a single MERGE statement.

 MERGE sync_publishers2 t USING @ids s ON s.[value] = t.sync_object_id and t.sync_registered_object_type_id = @objectTypeId WHEN MATCHED THEN UPDATE SET master_update_date = @date WHEN NOT MATCHED THEN INSERT (sync_object_id, sync_registered_object_type_id, master_update_date) VALUES (s.[value], @objectTypeId, @date);

Started getting deadlocks on the MERGE statements. The new deadlock graph can be viewed here: http://pastebin.com/QNJk7tea .

UPDATE 3. Trying MERGE hints

I've tried to do MERGE with the xlock and holdlock hints -- with no luck though -- got a deadlock on MERGE again.

MERGE sync_publishers2 with(xlock, holdlock) t

Answer 1

Here is the version that does not seem to deadlock after 1+ hour of 3 workload session in parallel. I can not really spot the exact reason of deadlocks in a first place, but what I can do is to stress out difference with deadlocking try that was also embracing the MERGE statement: the version bellow (that seems to work OK) uses CTE to allow MERGE 's ON expression to be rewritten in a way to only mention PK column ( sync_publisher_id ).

CREATE PROCEDURE [dbo].[SyncTracker_PublishEvent2]
    @objectTypeId int, 
    @ids dtInt readonly
AS
BEGIN
    SET NOCOUNT ON;

    -- stop recoursive propogations
    if(TRIGGER_NESTLEVEL() > 1) RETURN;

    declare @date datetime = getutcdate()

    ;WITH sync_publishers2CTE AS
    (
        SELECT [sync_publisher_id],
                [sync_object_id],
                [sync_registered_object_type_id],
                [master_update_date]
            FROM [dbo].[sync_publishers2] 
            WHERE sync_registered_object_type_id = @objectTypeId
    )
    MERGE sync_publishers2CTE WITH (XLOCK) trg
    USING 
    (
            SELECT sp.sync_publisher_id,
                    s.Value AS sync_object_id,
                    @objectTypeId AS sync_registered_object_type_id,
                    @date AS master_update_date
                FROM @ids s 
                LEFT JOIN sync_publishers2 sp ON sp.sync_object_id = s.Value
                                                AND sp.sync_registered_object_type_id = @objectTypeId 
    ) src
    ON (trg.sync_publisher_id = src.sync_publisher_id)
    WHEN MATCHED
        THEN UPDATE
            SET trg.master_update_date = src.master_update_date
    WHEN NOT MATCHED
        THEN INSERT
                 (sync_object_id, sync_registered_object_type_id, master_update_date)
            VALUES
                 (sync_object_id, sync_registered_object_type_id, master_update_date);
END

The sample execution plan:

Deadlocks after extracting part of the trigger into a stored procedure

Question

1 answers

solution1
0 2017-01-04 07:07:39

Deadlocks after extracting part of the trigger into a stored procedure

Question

1 answers

solution1 0 2017-01-04 07:07:39

solution1
0 2017-01-04 07:07:39