有没有比这更快的方法从T-SQL中的XML节点中提取数据？

Question

I am currently trying to create a stored procedure in T-SQL which takes an XML table as its input, and then inserts the data in it into a temporary table.

The XML that I am using has the following format:

<Table>
    <row MyFirstColumn="foo" MySecondColumn="bar" ... />
</Table>

The SQL that I am using to insert this XML data into a temporary table is of the following format:

INSERT INTO
    #TempTable
SELECT
    T.c.value('@MyFirstColumn', 'varchar(50)')
   ,T.c.value('@MySecondColumn', 'varchar(50)')
   ,...
FROM
    @x.nodes('//Table/row') T(c)

However, I am doing this with XML tables containing 150 columns and upwards of 200,000 rows. At present, executing this SQL on 10,000 rows takes ~142 seconds, so this is completely inappropriate for dealing with XML tables containing large numbers of rows.

Can anyone suggest a way to speed up this process?

Answer 1

Shredding XML with nodes()/value() in SQL Server has performance issues when you query a lot of columns. There is one nested loop join with a call to a xml function for each column.

Query plan with 3 columns:

Query plan with 5 columns:

Just imagine what it would look like with more than 150 columns.

Another option for you is to use OPENXML . It does not have the same problems with many columns.

Your query would look something like this:

declare @H int;
declare @X xml;

exec sys.sp_xml_preparedocument @H output,
                                @X;

select C1,
       C2,
       C3
from
       openxml(@H, 'Table/row', 0)
       with (
              C1 int,
              C2 int,
              C3 int
            );

exec sys.sp_xml_removedocument @H;

For me, using 150 columns and 1000 rows took about 14 seconds with nodes()/value() and 3 seconds with OPENXML.

Vote for a change.

Code used for testing;

drop table T;

go

declare @C int = 150;
declare @S nvarchar(max);
declare @X xml;
declare @N int = 1000;
declare @D datetime;

set @S = 'create table T('+
stuff((
      select top(@C) ', '+N'C'+cast(row_number() over(order by 1/0) as nvarchar(3)) + N' int'
      from sys.columns
      for xml path('')
      ), 1, 2, '') + ')'

exec sp_executesql @S;

set @S = 'insert into T select top(@N) '+
stuff((
      select top(@C) ',1'
      from sys.columns as c1
      for xml path('')
      ), 1, 1, '') + ' from sys.columns as c1, sys.columns as c2';

exec sp_executesql @S, N'@N int', @N;

set @X = (
         select *
         from dbo.T
         for xml raw, root('Table')
         );

set @S = 'select '+
stuff((
      select top(@C) ', '+N'T.X.value(''@C'+cast(row_number() over(order by 1/0) as nvarchar(3)) + N''', ''int'')'
      from sys.columns
      for xml path('')
      ), 1, 2, '') + ' from @X.nodes(''Table/row'') as T(X)'

set @D = getdate();
exec sp_executesql @S, N'@X xml', @X;
select datediff(second, @D, getdate());

set @S = 'declare @H int;
exec sp_xml_preparedocument @H output, @X;

select *
from openxml(@H, ''Table/row'', 0)
  with (' +
stuff((
      select top(@C) ', C'+cast(row_number() over(order by 1/0) as nvarchar(3))+ ' int'
      from sys.columns
      for xml path('')
      ), 1, 2, '') + ');
exec sys.sp_xml_removedocument @H';

set @D = getdate();
exec sp_executesql @S, N'@X xml', @X
select datediff(second, @D, getdate());

Answer 2

Your options depend on how much control you have over the server and what preparation you're willing and able to do.

If you have the ability to clean your data before calling the procedure (running an executable, for example)...

You could deserialize your data into an entity and use your ORM tool of choice (nHibernate, EntityFramework, etc.) to store the entity.

You could parse the XML into an object that a bulk importer could handle, store it to a file, and make use of sql's bulk import functionality. https://docs.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql?view=sql-server-2017

If you are able to make use of custom functionality on the server, you can use CLR user-defined functions to do this work instead of running it in a separate executable. https://docs.microsoft.com/en-us/sql/relational-databases/clr-integration-database-objects-user-defined-functions/clr-user-defined-functions?view=sql-server-2017

If I think of anything else I'll edit this post.

Answer 3

SQL-Server is pretty fast in dealing with XML, but you did not tell us the most important thing: Where is @x coming from?

Within SQL-Server the XML is not stored as as string you see, but as a hierarchically organised tree in physical tables. If you get this XML on string base and assign it to a variable of type XML , the engine will have to parse the whole lot and transfer all its content into the internal structures. The rest should be rather fast.

On the first sight there are two places to tune it a bit:

• FROM @x.nodes('//Table/row') T(c)
  The // will use a deep search , the engine will look into each <row> if there might be another <Table> nested below. Rather use FROM @x.nodes('/Table/row') T(c) .

• And use 'nvarchar(50)' instead of 'varchar(50)' . Internally XML stores its strings as NVARCHAR . You can avoid all these casts...

If you have SQL-Server 2016+ and you have control over the sender, you might give JSON a try. This is better in one-time-actions because it will not transfer your data in internal structures before it can work with it.

Answer 4

I really liked and voted for Mikael Eriksson answer, but there's one aspect about it:

His test generates 909 KB XML document with 1000 rows, 150 columns. And sp_xml_preparedocument takes only 226 milliseconds in his case (which is really fast), but...

I tried applying it to my XML document which is 521 MB. It contains 2045156 rows with 11 different columns, all are read as nvarchar(255)

When I selected all 11 columns via *:

• select * via .value() took 297 sec
• select * via openxml took 231 sec in total: (sp_xml_preparedocument took 107 sec, select * from openxml took 123 sec)

openxml works better in this case!

When I selected only 2 columns:

• select 2 columns via .value() took 57 sec
• select 2 columns via openxml took 189 sec in total: (sp_xml_preparedocument - 86 sec, select * from openxml - 103 sec)

.value() works better in this case!

So it looks like which method is faster actually depends on xml size, number of rows and number of columns that you query from xml!