DataTables, DataSets, Entity Framework, LINQ & Lambda Expression for Financial Technical Analysis C#

Question

this is more of an architectural question more than a specific code problem as I've hit a major block in how I am going to proceed with this project.

I'm building a financial scanning software that filters stock picks on specific criteria, for example. For example if out of 8000 stocks, its closing price today is above the SMA 100 and 10 days ago the closing price is below the SMA 100, then return this stock Symbol back to me.

However, note that the SMA (Simple Moving Average) is calculated with the last 100 days of data in the above example, but it could be that we could change the 100 days for lets say another value, 105 or 56 - could be anything.

In my Database I have a table definition called EODData with a few columns, here is the definition below;

EODData
sSymbol nvarchar(6)
mOpen money
mClose money
mHigh money
mLow money
Date datetime

The table will hold 3 years of End Of Day Data for the American Stock Exchange so that is approximately 6,264,000 rows, no problem for MS SQL 2008 R2.

Now, I'm currently using Entity Framework to retrieve data from my database, however what would be the best way to run or create my filter because the SMA must be calculated for each Symbol or underlying Stock Ticker each time a scan is performed because the 100 day variable can change.

Should I convert from Entity Objects to a DataSet for in memory filtering etc???

I've not worked with DataSets or DataTables much so I am looking for pointers.

Note that the SMA is just one of the filters, I have another algorithm that calculates the EMA (Exponential Moving Average, which is a much more complicated formula) and MACD (Moving Average Convergence Divergence).

Any opinions?

Answer 1

What about putting the calculations in the database? You have your EODData table, which is great. Create another table that is your SummaryData, something like:

SummaryData
stockSymbol varchar(6) -- don't need nvarchar, since AMSE doesn't have characters outside of normal English alphabet.
SMA decimal
MCDA decimal
EMA decimal

Then you can write some stored procedures that run on close of day and update this one table based on the data in your EODData table. Or you could write a trigger so that each insert of the EODData table updates the summary data in your database.

One downside to this is that you're putting some business logic in the database. Another downside is that you will be updating statistical data on a stock symbol that you might not need to do. For example, if nobody ever wants to see what XYZZ did, then the calculation on it is pointless.

However, this second issue is mitigated by the fact that 1. you're running SPs on the server which MSSQL can optimize and 2. You can run these after hours when everyone is at home, so if it takes a little bit of time you're not affected. (To be honest, I'd assume if they're calculations like rolling averages, min/max etc, SQL won't be that slow.)

One upside is your queries should be wicked fast, because you could write

select stockSymbol from SummaryData where SMA > 10 you've already done the calculation.

Another upside is that the data will only change once per day (at the close of the business day) but you might be querying several times throughout the day. For example, you want to run several different queries today for all the data up to and including yesterday. If you run 10 queries, and your partner runs the same 10 queries, you've just done the same calculation over. (Essentially, write once, read many.)