SQL: xml.nodes from cte are very slowly

Question

I have table with a column that contains xml like this:

<block>
    <blockIn>
        <G>1</G>            
    </blockIn>
    .....
    <blockIn>
        <G>12</G>
    </blockIn>
    ......
</block>
.....
<block>
......
</block>

I need find MAX between <blockIn><G> in each , and then summarize all this MAX

(sum (Max (<block> …<blockIn> ...<G></G>); Max (<block> …<blockIn> ...<G></G>) ...))

So, I did this:

WITH ds AS 
(
    SELECT 
        fieldXML
    FROM 
        table
    WHERE 
        ID = 1
)
SELECT 
    (SELECT SUM(node_a.value('max(blockIn/G)' , 'int' )) 
     FROM ds.fieldXML.nodes('/Block')  AS node_refs(node_a)) AS [ArticulNum]
FROM
    ds

But it works very slowly.

If I use a variable, it works very fast:

DECLARE @xml AS [XML];

SELECT 
    @xml = fieldXML
FROM 
    table
WHERE 
    ID = 1;

SELECT SUM(node_a.value('max(blockIn/G)' , 'INT' )) 
FROM @xml.fieldXML.nodes('/Block') AS node_refs(node_a)

What do I need to do so that the first solution works fast, too?

Answer 1

A User Defined Function (UDF) will help but it needs to be the right kind of UDF, and, if performance is important, then it must be an Inline Function . Here's a cleaned up version of your original (note that the final SUM is not required):

-- Original
CREATE OR ALTER FUNCTION [dbo].[ArticulNumFromXML_original](@xml XML)  
RETURNS INT
AS  
BEGIN  
RETURN 
(
  SELECT node_a.value('max(blockIn/G)' , 'int' )
  FROM   @xml.nodes('/block') AS node_refs(node_a)
    ); 
END;  
GO

Here's an improved scalar UDF that will perform better. Note the different context block/blockIn and the use of the text() node.

-- Improved scalar UDF:
CREATE OR ALTER FUNCTION [dbo].[ArticulNumFromXML_V2](@xml XML)  
RETURNS INT  
AS  
BEGIN  
RETURN 
(
  SELECT Mx = MAX(node_a.value('(G/text())[1]','int'))
  FROM   @xml.nodes('/block/blockIn') AS node_refs(node_a)
    ); 
END;  
GO

This will perform much better but still has a fundamental problem: the function is not inline. Let's take the logic above to create an inline table valued function (iTVF):

-- INLINE UDF    
CREATE OR ALTER FUNCTION [dbo].[ArticulNumFromXML_itvf](@xml XML)  
RETURNS TABLE AS RETURN 
  SELECT Mx = MAX(node_a.value('(G/text())[1]','int'))
  FROM   @xml.nodes('/block/blockIn') AS node_refs(node_a);
GO

Next for a sample xml data generator for performance testing. This code will create a table with 20K random XML values:

IF OBJECT_ID('tempdb..#yourtable') IS NOT NULL DROP TABLE #yourtable;
SELECT TOP (20000) 
  SomeId  = IDENTITY(INT,1,1),
  xmldata = CAST(f.X AS XML),
  blob    = CAST(CAST(f.X AS VARBINARY(MAX)) AS image) 
INTO #yourtable
FROM       (VALUES (1),(1),(1),(1),(1),(1),(1),(1),(1),(1)) AS a(X) -- 10
CROSS JOIN (VALUES (1),(1),(1),(1),(1),(1),(1),(1),(1),(1)) AS b(X) -- 100
CROSS JOIN (VALUES (1),(1),(1),(1),(1),(1),(1),(1),(1),(1)) AS c(X) -- 1K
CROSS JOIN (VALUES (1),(1),(1),(1),(1),(1),(1),(1),(1),(1)) AS d(X) -- 10K
CROSS JOIN (VALUES (1),(1),(1),(1),(1),(1),(1),(1),(1),(1)) AS e(X) -- 100K
CROSS JOIN (VALUES (NEWID())) AS n(Id)
CROSS APPLY
(
  SELECT TOP(ABS(CHECKSUM(NEWID())%5)+b.X) 
     G = ABS(CHECKSUM(n.Id)%30)+c.X+ROW_NUMBER() OVER (ORDER BY (SELECT 1))
  FROM (VALUES (1),(1),(1),(1),(1),(1),(1),(1),(1),(1)) AS a(x)
  ORDER BY NEWID()
  FOR XML PATH('blockIn'), ROOT('block')
) AS f(x);

Next for a quick sanity check. The queries below will return the same results:

-- Sanity Check (all 3 return the same results)
SELECT TOP (10) t.SomeId, Mx = dbo.ArticulNumFromXML_original(xmldata)
FROM     #yourtable AS t
ORDER BY t.SomeId;

SELECT TOP (10) t.SomeId, Mx = dbo.ArticulNumFromXML_V2(xmldata)
FROM   #yourtable AS t
ORDER BY t.SomeId;

SELECT TOP (10) t.SomeId, f.Mx
FROM        #yourtable AS t
CROSS APPLY dbo.ArticulNumFromXML_itvf(xmldata) AS f
ORDER BY t.SomeId;

Now that we know we're getting the right result set let's do a couple performance tests. I noticed that, in your answer, you're converting the XML data first. This is expensive. In this first test I'm we're doing the same type of conversion:

-- Test #1: Blob data
PRINT CHAR(13)+'Scalar Version (original):'+CHAR(13)+REPLICATE('-',90);
GO
DECLARE @st DATETIME = getdate(), @Mx INT;
  SELECT @Mx = dbo.ArticulNumFromXML_original(CAST(CAST(t.blob AS VARBINARY(MAX)) AS XML))
  FROM   #yourtable AS t;
PRINT DATEDIFF(MS,@st,getdate());
GO 3

PRINT CHAR(13)+'Scalar Version (V2 - leveraging the text() node):'+CHAR(13)+REPLICATE('-',90);
GO
DECLARE @st DATETIME = getdate(), @Mx INT;
  SELECT @Mx = dbo.ArticulNumFromXML_V2(CAST(CAST(t.blob AS VARBINARY(MAX)) AS XML))
  FROM   #yourtable AS t;
PRINT DATEDIFF(MS,@st,getdate());
GO 3

PRINT CHAR(13)+'Inline Version:'+CHAR(13)+REPLICATE('-',90);
GO
DECLARE @st DATETIME = getdate(), @Mx INT;
  SELECT      @Mx = f.Mx
  FROM        #yourtable AS t
  CROSS APPLY dbo.ArticulNumFromXML_itvf(CAST(CAST(t.blob AS VARBINARY(MAX)) AS XML)) AS f;
PRINT DATEDIFF(MS,@st,getdate());
GO 3

Results:

Scalar Version (original):
------------------------------------------------------------------------------------------
Beginning execution loop
4560
4000
4346
Batch execution completed 3 times.

Scalar Version (V2 - leveraging the text() node):
------------------------------------------------------------------------------------------
Beginning execution loop
2503
2840
2796
Batch execution completed 3 times.

Inline Version:
------------------------------------------------------------------------------------------
Beginning execution loop
586
670
630
Batch execution completed 3 times.

As you can see: The first improvement sped things up better than 50% but, changing the function to an Inline Table Value Function made the improved query 5-6 times faster and almost 10 times faster than your original function.

Now let's skip the costly XML conversion (this can be handled via pre-processing using a computed column or indexed view. Here's the second test:

-- Test #2: No XML Conversion
PRINT CHAR(13)+'Scalar Version (original):'+CHAR(13)+REPLICATE('-',90);
GO
DECLARE @st DATETIME = getdate(), @Mx INT;
  SELECT @Mx = dbo.ArticulNumFromXML_original(xmldata)
  FROM   #yourtable AS t;
PRINT DATEDIFF(MS,@st,getdate());
GO 3

PRINT CHAR(13)+'Scalar Version (V2 - leveraging the text() node):'+CHAR(13)+REPLICATE('-',90);
GO
DECLARE @st DATETIME = getdate(), @Mx INT;
  SELECT @Mx = dbo.ArticulNumFromXML_V2(xmldata)
  FROM   #yourtable AS t;
PRINT DATEDIFF(MS,@st,getdate());
GO 3

PRINT CHAR(13)+'Inline Version (No hints - Parallel):'+CHAR(13)+REPLICATE('-',90);
GO
DECLARE @st DATETIME = getdate(), @Mx INT;
  SELECT      @Mx = f.Mx
  FROM        #yourtable AS t
  CROSS APPLY dbo.ArticulNumFromXML_itvf(xmldata) AS f;
PRINT DATEDIFF(MS,@st,getdate());
GO 3

Results:

Scalar Version (original):
------------------------------------------------------------------------------------------
Beginning execution loop
2933
2633
2953
Batch execution completed 3 times.

Scalar Version:
------------------------------------------------------------------------------------------
Beginning execution loop
826
876
970
Batch execution completed 3 times.

Inline Version (No hints - Parallel):
------------------------------------------------------------------------------------------
Beginning execution loop
63
70
63
Batch execution completed 3 times.

Blam! reading pre-converted XML reduced the time of all three dramatically, more so for the iTVF which is now 40-50 times faster than your original function.

Answer 2

decision, made function:

CREATE OR ALTER FUNCTION [dbo].[ArticulNumFromXML](@xml XML)  
RETURNS INT  
AS  
BEGIN  
RETURN (SELECT 
      SUM(node_a.value('max(blockIn/G)' , 'int' )) 
    FROM 
      @xml.nodes('/BLOCK') AS node_refs(node_a)
    ); 
END;  
GO

and with it, normal:

SELECT 
  [dbo].[ArticulNumFromXML](CAST(CAST(blob AS VARBINARY(max)) AS XML)) 
FROM 
  table
WHERE 
  ID = 1