Rounding up of data in SQL Server 2014

Question

I have developed an application in C# (ASP.NET 4.0) using SQL Server 2014 as the database.

I have question about rounding up and summing data. I have data that comes in via CSV through FTP and I import the raw data into a table. The data is logged every minute by the customer. The data is identified by customer Id.

I have now been asked to take that data and sum the time series data into 15 minutes chunks from the hour.

They then want that data rounded up into days (midnight to midnight), then that data rounded up into weeks (Monday to Sunday). They also want the day data rounded up into calendar months from midnight to midnight and the month data is to be rounded up into a year.

The idea is that the raw time series data is grouped into its constituent periods, such as day, week, month so they can see the total for that time period.

I have looked at cursors and loops in SQL and I have been told the overhead will be too great as I already have 300 million rows and counting to process.

I don't whether I should develop a service in C# that does it all on the server or use the database. The research I have done contradicts itself slightly in each case.

Any hints would be great as where to look and what to try.

Answer 1

I believe you are looking more for a desighn than a solution here. I would suggest you to create a table which will hold the data of the ftp load along with a batch id (a unique identifier). create another table where you load this batch id with a status column, alays insert a row here once if you are doen with ftp load into the table1, make status as N. This polling script should call the below sp.

Now, create a polling script from c# or if you are experienced with service broker in sql use that, to poll this table2 with batch id and status with status as N.

Now, create another stored procedure which will sum up all the records for this batch id only. And add the values to the daily count approaitely.. The same will be done weekly counts and all...

Once all this is done, remove the information from the table1 with the batch id for which we processed, if you need this info for future purpose you can sotre it in different table.

Answer 2

To have the power of managing data and ready for any change for business rules in the future, you need to add some control columns to the table. The controls managing period/hour/day/month/year /... whatever in future

Just when you add the period fill the corresponding control fields once at a time with the corresponding value:

period 1..4
hour 1-24
day 1..366
week 1..55
month 1..12
year 1.. (if needed)

You can define set of SQL functions to fill these columns at once(during data loading from the file).

Create index for these columns.

Once , you do this you can by, c# code /sql code, you do the summing up dynamically to any period/hour/day/..... You can benefit from Analysis server / window functions / pivots to do your magic :) on data for any interval.

This approach gives you the power of keeping data , no deletion , except for archiving purpose and managing changes in the future.