遍历大型数据集时要在内存中保留多少数据

Question

I am trying to create a trading simulator to test strategies over long periods of time.

I am using 1 minute data points. So if I were to run a simulation for say 10 years that would be approx 3,740,000 prices (Price class shown below). A simulation could be much longer than 10 years but using this an example.

     class Price
     {
          DateTime DatePrice
          double Open
          double High
          double Low
          double Close
     }

My simulator works however I can't help feeling the way I'm doing it isn't very optimal.

So currently what I do is to grab a years worth of prices from my SQL database so approx 374,400 prices. I do this because I don't want to use too much memory (this might be misguided have no idea).

Now when looping through time the code will also make use of the previous lets say 10 prices. So at 2.30am the code will look back the prices from 2.20am, all the prices previous to this are now redundant. So it seems somewhat wasteful to me if I hold 374,400 prices in memory.

            time       Close
            00:00      102
            00:01      99
            00:02      100
            ...
            02:20      84
            02:21      88

So I have a loop that will loop through from my start date to my end date, checking at each step if I need to download additional prices from the database.

   List<Price> PriceList = Database.GetPrices(first years worth or prices) 

   for(DateTime dtNow = dtStart; dtNow < dtEnd; dtNow = dtNow.AddMinutes(1))
   {
         // run some calculations which doesn't take long

         // then check if PriceList[i] == PriceList.Count - 1
         // if so get more prices from the database and obviously reset i to zero but baring in mind I need to keep the previous 10 prices
   }

What is the best solution for this kind of problem? Should I be getting prices from the database on another thread or something?

Answer 1

Lets do some math

class Price
{
    DateTime DatePrice;
    double Open;
    double High;
    double Low;
    double Close;
}

has a size of 8(DateTime)+4*8(double) = 40 alone for the members. Since it is a reference type you need a method table pointer and a SyncBlock pointer which add 16 byte additionally. Since you need to keep the pointer to the object (8 bytes on x64) somewhere we get a total size per instance of 64 bytes.

If you want to have a 10 year history with 3,7 million instances you will need 237 MB of memory which is not much in todays world.

You can shave off some overhead by switching from double to floats which will need only 4 bytes and if you go with a struct

struct Price
{
    DateTime DatePrice;
    float Open;
    float High;
    float Low;
    float Close;
}

You will need only 24 bytes with no big loss of precision since the value range of stocks are not so high and you are interested in a long term trend or pattern and not 0,000000x fractions.

With this struct your 10 year time horizon will cost you only 88MB and it will keep the garbage collector off your data because it is opaque for the GC (no reference types inside your struct).

That simple optimization should be good enough for time horizons which span hundreds of years even with todays computers and memory sizes. It would even fit into an x86 address space but I would recommend running this on x64 because I suspect you will check not only one stock but several ones in parallel.

Answer 2

If I were you, I would keep the problem of caching (which seems to be your problem), separate from functionality.

I don't know how you currently fetch your data from the DB. I am guessing you are using some logic similar to

DataAdapter.Fill(dataset);
List<Price> PriceList = dataset.Tables[0].SomeLinqQuery();

Instead of fetching all the prices at teh same time, you can use something like below to fetch them incrementally and convert the fetched row into a Price object

IDataReader rdr = IDbCommand.ExecuteReader();
while(rdr.Read())
{
}

Now to make transparent access to Prices, you might want to roll in some class which can provide caching

class FixedSizeCircularBuffer<T> {
    void Add(T item) { } // make sure to dequeue automatically to keep buffer size fixed
    public T GetPrevious(int relativePosition) { } // provide indexer to access relative to the current element
}

class CachedPrices {
    FixedSizeCircularBuffer<Price> Cache;

    public CachedPrices() {
        // connect to the DB and do ExecuteReader
        // set the cache object to a good size
    }

    public Price this[int i] {
        get {
            if (i is in Cache)
                return Cache[i];
            else
                reader.Read();
                //store the newly fetched item to cache
        }
    }

}

Once you have such infrastructure, then you can pretty much use it to restrict how much pricing information is loaded and keep your functionality separate from the Caching mechanism. This provides you the flexibility to control how much memory you have to spare for pre-fetching prices and the amount of data you can process

Needless to say, this is just a guideline - you will have to understand this and implement for yourself

Answer 3

From a time efficiency perspective, what would be optimal is for you to get back an initial batch of prices, start processing those, then immediately begin to retrieve the rest. The problem with checking for new data during your processing is that you have to delay your program everytime you need new data.

If you really do care about memory, what you need to do is remove prices from the list you have after you are done with them. This will allow the garbage collector to free up the consumed memory. Otherwise with what you have, once your program is finishing and you pulled back the last year of prices you will have retrieved all of the prices and you would be consuming as much memory as if you had gotten all of the prices at once.

I believe you are being premature with your memory concerns. The only time I ever had to worry about memory/the garbage collector in .net was when I had a long running process and one step in that process included downloading PDF's. Even though I retrieved the PDF's as needed, the PDF's in memory would eventually consume GB's of memory after running for a while and throw an exception after consuming whatever the .net memory limit is for lists.