Recursion Help required with C#

Question

On our windows application, We have startDate and EndDate. On click of Execute button event, we need to call a third party web service with our search string + daterange( date from 01/01/2010 to 12/31/2010). Now our search criteria can return us thousands of records but web service have limitation of able to return only 10K records per transaction.

Which required us to break down our dateRange. So basically we need following;

For (X dateRange if RecordCount > 10000) then X dateRange/2 which will be 01/01/2010 to 06/01/2010 in our case and check condition again and do this recursively until we get daterange block where RecordCount is < 10000.

Then start with Next date, for example, if we get 9999 records for 01/01/2010 to 03/30/2010 then we need to get records for next block starting 04/01/2010

Is this possible with Recursion?

RecursionFunction(dtStart, dtEnd)
{
  if (WebService.RecordCount > 9999)
     {
                TimeSpan timeSpan = dtEnd.Subtract(dtStart);
                DateTime mStart = dtStart;
                DateTime mEnd = dtStart.AddDays(timeSpan.Days / 2);
                RecursionFunction(dtStart,dtEnd);
      }
  else
   {
     Get Records here
    }
}

But with above code, recursion will have following blocks

01/01/2010, 12/31/2010 > 10000 01/01/2010, 07/03/2010 > 10000 01/01/2010, 04/02/2010 < 10000

So after finishing getting record, recursion will start again with block 01/01/2010,07/03/2010 which we don't need. We need to start next recursion with 04/03/2010,12/31/2010

Thanks in advance for help.

Answer 1

The first step is to change the RecursionFunction call (at line 8 of your example) to:

RecursionFunction(mStart, mEnd);

But, then, you'll also need to call it again with the other half of the date range.

RecursionFunction(mEnd + AddDays(1), dtEnd);

Also, you need to handle the results (presumably combining the two answers).

var set1 = RecurseFunction(...);
var set2 = RecurseFunction(...);
return set1.Concat(set2);

Answer 2

It looks like you are trying to split the input range until it is small enough to handle. Try calling it for both ranges:

RecursionFunction(mStart, mEnd);
RecursionFunction(mEnd.AddDays(1), dtEnd);

Answer 3

This is like divide and conquer. You need to get results from the left and the right of the split and combine them and return that value. So you can keep getting smaller until you have enough data you can deal with and just return that. Then keep joining the result sets together.

public IList<Data> GetRecords(DateTime start, DateTime end)
{
    var RecordCount = WebService.RecordCount(start, end);
    if (RecordCount < 10000) return WebService.GetRecords(start, end);
    DateTime l, m, e;
    l = start;
    e = end;
    var midDay = end.Subtract(start).TotalDays / 2;
    m = start.AddDays(midDay);

    var left = GetRecords(l, m);
    var right = GetRecords(m.AddDays(1), e);

    return left.Concat(right);
}

Answer 4

An easy way would be a form of pagination. If your using JSON or XML, you can put the amount of total results and just return a set number of results (return the offset too). This way you can do a loop to check if your on the last page and after you get the last results page, break out of it.

Don't forget to put checks in if a particular transaction fails though. It's not an ideal solution on such a large dataset but it is a workaround

Answer 5

It sounds much easier to just reuse the last date for the data you actually got back in a while-loop than to home in with recursion like this.

Then start with Next date, for example, if we get 9999 records for 01/01/2010 to 03/30/2010 then we need to get records for next block starting 04/01/2010

March has 31 days.

Pseudo-C# code

var dtStart = DateTime.Parse("2010-01-01");
var dtEnd = DateTime.Parse("2010-12-31");

var totalRecords = new List<RecordType>();
var records = WebService.Get(dtStart, dtEnd);
totalRecords.Add(records);
while (dtStart < dtEnd && records.Count > 9999)
{
    dtStart=records.Last().Date;
    records = WebService.Get(dtStart, dtEnd);
    totalRecords.Add(records);
}

To ease the load on the service you could calculate the timespan for the previous run and only get that many days for the next run in the while-loop.

How you should handle the inevitable doublets depends on the data on the records.

I just realized I presumed you had a date in the returned data. If not, then disregard this answer.

Answer 6

This is how I would do it

static List<string> RecursiveGet(DateTime StartDate, DateTime EndDate, List<string> Output)
{
    if (Webservice.RecordCount > 9999)
    {
        TimeSpan T = EndDate.Subtract(StartDate);
        T = new TimeSpan((long)(T.Ticks / 2));
        DateTime MidDate = StartDate.Add(T);
        Output.AddRange(RecursiveGet(StartDate, MidDate, Output));
        Output.AddRange(RecursiveGet(MidDate.AddMilliseconds(1), EndDate, Output));
    }
    else
    {
        //Get Records here, return them in array
        Output.Add("Test");
    }
    return Output;
}
static List<string> GetRecords(DateTime StartDate, DateTime EndDate)
{
    return RecursiveGet (StartDate, EndDate, new List<string>());
}

Note, Couldn't test it
It works by dividing the dates in half, then searching each of them, and if one is still bigger than 9999, then doing it again.