C# 中的数据处理 - 最佳方法？

Question

I am still on a learning curve in C# and SQL Server so please forgive my 'greeness'.

Here is my scenario:

I have an EMPLOYEE table with 10,000 rows. Each of these employees has transactions in a TRANSACTIONS table.

The transaction table has the salary elements like Basic pay, Acting allowance, Overtime hours etc. It also has payroll deductions like advance deductions, some loans (with interest), and savings (pension, social security savings etc.

I need to go through each employee's transactions and compute taxes, outstanding balances on loans, update balances on savings, convert hours into payments/deductions and some other stuff.

This processing will give me a new set of rows for each employee, with a period marker (eg 2013-04 for April 2013). I need to store this in a HISTORY table for future references.

What is the best approach for processing the entire 10,000 employee table and their transactions?

I am told that pulling the entire table into memory via readers is not good practice and I agree.

Do I keep pulling an employee from the database, process their transactions, and commit the history to the database? And pull the next and so forth?

Too many calls to the back end?

(EF not an option for me, still doing raw SQL in ADO.NET)

I will appreciate any help on this.

Answer 1

10000 rows is not much. Memory could easily handle that if there's not some enourmous varchar or binary columns. Don't feel completely locked by good practice "rules".

On the other hand, consider a stored procedure . Then all processing will be done locally on the server.

edit: if neither of the above is an option, try to stream your results. For example, when reading your query save each row in a ConcurrentQueue or something like that. Before you execute the query, start another thread or a BackgroundWorker which checks the queue for new items and saves back results simultaneously on another SqlConnection . Work will be done when query is done AND the queue has Count 0.

Answer 2

Check out using ROW_NUMBER() . This can be used by programs to allow large tables to be essentially browsed using 'x' number of rows at a time. You could then conceivably use this same method to batch your job over, say, 1000 rows at a time.

See this link for more information.