分布式并行批处理

Question

I am trying to figure out a solution for a problem statement. For simplicity's sake the application is going to be a console app. A database contains some 10000 rows. I have a to develop ac# application which will spawn out 10 threads and each thread will process 5 records from the database. Now the thing is i will run the same application in another machine. So we have two instance of he application and each application should process 5000 rows and they should not overlap on the records. Also i need to have a central controller which needs to display the thread ids that are running on each application. Any pointers here to solve this problem would be great.

Answer 1

I would create a 'Processing Status' column in the DB and each instance will mark the records it's processing by changing record's status.

Another solution is to use a queue or Service Broker and each instance would be picking a message to process a record from common queue. You will need another process here that will populate the queue.

Using WCF to communicate between machines to synchronize work is IMHO too complex and tricky to get right.

Answer 2

I agree with Jakub. If you want the separate instances to communicate with each other directly you should read about the Two Generals problem (http://en.wikipedia.org/wiki/Two_Generals%27_Problem) and think about how you're going to design a non blocking rendezvous without races.