insert records in database from an xml

Question

Am writing a .net windows service using which i need to parse an xml (having about 5000 nodes). I need to parse these nodes and insert data (ie. 5000 rows)into a sql database. Shall I insert all these records with batch insert or shall i insert them one by one? can someone help me with the design/algorithm for optimum performance?

Answer 1

From Importing XML data into a SQL Server table with C# asp.net :

DataSet reportData = new DataSet();
reportData.ReadXml(Server.MapPath(”report.xml”));

SqlConnection connection = new SqlConnection(”CONNECTION STRING”);
SqlBulkCopy sbc = new SqlBulkCopy(connection);
sbc.DestinationTableName = “report_table”;

//if your DB col names don’t match your XML element names 100%
//then relate the source XML elements (1st param) with the destination DB cols
sbc.ColumnMappings.Add(”campaign”, “campaign_id”);
sbc.ColumnMappings.Add(”cost”, “cost_USD”);

connection.Open();

//table 4 is the main table in this dataset
sbc.WriteToServer(reportData.Tables[4]);
connection.Close();

Answer 2

Since you are already using SQL Server, I would recommend checking out SSIS(Integration Services). It has a built in XML Source that will pump right into SQL Server. It is fast and would save you from maintaining this code.

Answer 3

For optimum performance use a forward only XML reader , a transaction and a bulk inserter

Of course, this solution is slightly trickier and its quite possible a simpler implementation exists that requires less code and performs pretty well (EG. OPENXML )

Answer 4

You could use a XSLT transformation that transforms the XML into a SQL file with insert statements.

I did that once as a proof of concept in a day or so. Here's what I did:

<xsl:strip-space elements="*"/>

<xsl:template match="auditlog">
<xsl:if test="sequence/struct/wstring[@value='edr']/../union/string/@value">
INSERT INTO AUDITLOGS (FILENAME, EVENTTIME, EDR) VALUES(
  '<xsl:value-of select="@filename"/>',
  '<xsl:value-of select="sequence/struct/union/any/sequence/struct/struct[@name='m_eventTime']/@datetime"/>',
  '<xsl:value-of select="sequence/struct/wstring[@value='edr']/../union/string/@value"/>'
);
<xsl:apply-templates select="sequence"/>
</xsl:if>
</xsl:template>

<xsl:template match="struct[@id='ModifiedBalance']">
INSERT INTO MODIFIEDBALANCE (SOURCEKIND, SOURCEID, TYPEID, NAMEID, ORIGINAL, CURRENT, LOGID)
SELECT
  '<xsl:value-of select="struct/enum/@value"/>',
  <xsl:value-of select="struct/longlong/@value"/>,
  <xsl:value-of select="struct/ushort[@id='BalanceTypeId']/@value"/>,
  <xsl:value-of select="struct/ushort[@id='BalanceNameId']/@value"/>,
  <xsl:value-of select="struct[@name='m_original']/@amount"/>,
  <xsl:value-of select="struct[@name='m_current']/@amount"/>,
  LOGID
FROM AUDITLOGS
  WHERE FILENAME = '<xsl:value-of select="../../../../../@filename"/>';
</xsl:template>

</xsl:stylesheet>