Loading large amounts of data into a List collection of business objects from Oracle

Question

I need a way to optimize the data retrieval of millions of records from an Oracle database into a List of custom business object. The data that is being returned from Oracle is in XML format and I need a way to serialize it into a List of business objects.

The code I wrote is working properly, however, it is taking a long time to execute during the loading of the XML into memory, specifically when the code hits the line:

var xDoc = XDocument.Load(xmlReader);   

Code :

//Custom Business object  
 public class AccountType
    {
        public int AccountTypeID { get; set; }
        public string AccountCode { get; set; }
        public string BookType { get; set; }
        public int Status { get; set; }
    }

//Code that retrieves data from Oracle DB

  using (OracleConnection objOracleConnection = new OracleConnection(strConnectionString))
            {
                using (OracleCommand orclCmd = objOracleConnection.CreateCommand())
                {
                    try
                    {

                        orclCmd.CommandText = strXMLSQL;
                        orclCmd.BindByName = true;
                        orclCmd.XmlCommandType = OracleXmlCommandType.Query;
                        orclCmd.XmlQueryProperties.RootTag = "AccountData";
                        orclCmd.XmlQueryProperties.RowTag = "ROW";

                        objOracleConnection.Open();
                        XmlReader xmlReader = orclCmd.ExecuteXmlReader();
                        var xDoc = XDocument.Load(xmlReader);                             
                        List<AccountType> accountTypes = (from data in xDoc.Root.Elements("ROW")
                                                              select new AccountType
                                                              {
                                                                  AccountTypeID = data.GetIntXMLElementValue("ACCOUNTTYPEID"),
                                                                  AccountCode = data.GetStringXMLElementValue("ACCOUNTCODE"),
                                                                  BookType = data.GetStringXMLElementValue("BOOKTYPE"),
                                                                  Status = data.GetIntXMLElementValue("STATUS")
                                                              }).ToList();

                    }
                    catch (OracleException oracleEx)
                    {
                        throw oracleEx;
                    }
                    catch (Exception generalEx)
                    {
                        throw generalEx;
                    }
                    finally
                    {
                        objOracleConnection.Close();
                    }
 }

Any help would be greatly appreciated.

Thanks!

Answer 1

Do you need the millions of records all at once?

Is the data stored in the database as XML? If not you can use ExexcutePageReader instead of ExecuteXmlReader

If it has to be XML then you could implement your own pager by calling for records where id is between lastId plus 1 and lastId plus pageLength

Answer 2

Paging is an option. Also threading using PLINQ right before you call the ToList() because that is where you are taking the hit. I would also add that removing the XDocument loading could help. Try deserializing the xml. Assuming your XML looks like this

<ROWSET>
    <ROW>
        <AccountData>
            <ACCOUNTTYPEID>1</ACCOUNTTYPEID>
            <ACCOUNTCODE>ABC</ACCOUNTCODE>
            <BOOKTYPE>FOO</BOOKTYPE>
            <STATUS>10</STATUS>
        </AccountData>
    </ROW>
    <ROW>
        <AccountData>
            <ACCOUNTTYPEID>2</ACCOUNTTYPEID>
            <ACCOUNTCODE>XYZ</ACCOUNTCODE>
            <BOOKTYPE>BAR</BOOKTYPE>
            <STATUS>20</STATUS>
        </AccountData>
    </ROW>
</ROWSET>

You can set up the following contracts:

[DataContract(Namespace = "")]
public class AccountData
{
    [DataMember(Name = "ACCOUNTTYPEID")]
    public int Id { get; set; }
}

[DataContract(Name = "ROW", Namespace = "")]
public class Row
{
    [DataMember(Name = "AccountData")]
    public AccountData Data { get; set; }
}

[CollectionDataContract(Name="ROWSET", Namespace = "")]
public class RowSet
    : List<Row>
{
}

and de-serialize like this

var s = new DataContractSerializer(typeof(RowSet));
var o = s.ReadObject(xmlreader) as RowSet;

This avoides the XDocument loading and LINQ-XML overhead.