C# Out of Memory retrieving entity details from Dynamics 365

Question

Since I have millions of records in Dynamics 365 and I need to retrieve them all, I am using paging to retrieve the entity data:

// Query using the paging cookie.
// Define the paging attributes.
// The number of records per page to retrieve.
int queryCount = 5000;

// Initialize the page number.
int pageNumber = 1;

// Initialize the number of records.
int recordCount = 0;

// Define the condition expression for retrieving records.
ConditionExpression pagecondition = new ConditionExpression();
pagecondition.AttributeName = "parentaccountid";
pagecondition.Operator = ConditionOperator.Equal;
pagecondition.Values.Add(_parentAccountId);

// Define the order expression to retrieve the records.
OrderExpression order = new OrderExpression();
order.AttributeName = "name";
order.OrderType = OrderType.Ascending;

// Create the query expression and add condition.
QueryExpression pagequery = new QueryExpression();
pagequery.EntityName = "account";
pagequery.Criteria.AddCondition(pagecondition);
pagequery.Orders.Add(order);
pagequery.ColumnSet.AddColumns("name", "emailaddress1");                   

// Assign the pageinfo properties to the query expression.
pagequery.PageInfo = new PagingInfo();
pagequery.PageInfo.Count = queryCount;
pagequery.PageInfo.PageNumber = pageNumber;

// The current paging cookie. When retrieving the first page, 
// pagingCookie should be null.
pagequery.PageInfo.PagingCookie = null;
Console.WriteLine("Retrieving sample account records in pages...\n");
Console.WriteLine("#\tAccount Name\t\tEmail Address"); 

while (true)
{
    // Retrieve the page.
    EntityCollection results = _serviceProxy.RetrieveMultiple(pagequery);
    if (results.Entities != null)
    {
        // Retrieve all records from the result set.
        foreach (Account acct in results.Entities)
        {
            Console.WriteLine("{0}.\t{1}\t{2}", ++recordCount, acct.Name,
                               acct.EMailAddress1);
        }
    }

    // Check for more records, if it returns true.
    if (results.MoreRecords)
    {
        Console.WriteLine("\n****************\nPage number {0}\n****************", pagequery.PageInfo.PageNumber);
        Console.WriteLine("#\tAccount Name\t\tEmail Address");

        // Increment the page number to retrieve the next page.
        pagequery.PageInfo.PageNumber++;
        
        // Set the paging cookie to the paging cookie returned from current results.
        pagequery.PageInfo.PagingCookie = results.PagingCookie;
    }
    else
    {
        // If no more records are in the result nodes, exit the loop.
        break;
    }
}

After a couple of hours of pulling data, the program ends saying OutofMemory exception. It would appear that memory usage grows with every new page. How does one clear the memory usage to avoid this problem?

Answer 1

It's good that you are retrieving only a couple columns. Getting all the columns would certainly increase memory usage.

They way that you're processing each page it would seem that the runtime should release each results object after you output it to the console. It seems that something might be causing each results objects to stay on the heap.

Maybe it's the reuse of the same pagequery object.

The only thing that looks like it persists from the results is the paging cookie.

pagequery.PageInfo.PagingCookie = results.PagingCookie;

Maybe try moving the creation of the QueryExpression into a method and create a new one for each page. For example:

private QueryExpression getQuery(int pageNum, string pagingCookie)
{
    //all query creation code
}

What is going on with the condition for the parentaccountid ? Does a single parent have millions of child accounts?

You might also want to consider tweaking the while loop:

var continue = true;
while(continue)
{
    //process
    continue = results.MoreRecords;
}