How to manage multithread c# application

Question

I have following scenario:

C# application (.net 4.0/4.5), with 5-6 different threads. Every thread has a different task, which is launched every x seconds (ranging from 5 to 300).

Each task has following steps:

1. Fetch items from Sql Server
2. Convert items in Json
3. Send data to webserver
4. Wait for reply from server.

Since this tasks can fail at some point (internet problems, timeout, etc) what is best solution in .NET world?

I thought about following solutions:

• Spawn new thread every x seconds (if there is not another thread of this type in execution)
• Spawn one thread for each type of task and loop steps every x seconds (to understand the way to manage exceptions)

Which would be more secure and robust? Application will run on unattended systems, so it should be able to remain in execution regardless of any possible exception.

Answer 1

Threads are pretty expensive to create. The first option isn't a great one. If the cycle of "do stuff" is pretty brief (between the pauses), you might consider using the ThreadPool or the TPL. If the threads are mostly busy, or the work takes any appreciable time, then dedicated workers are more appropriate.

As for exceptions: don't let exceptions escape workers. You must catch them. If all that means is that you give up and retry in a few seconds, that is probably fine.

Answer 2

You could have modeled the whole thing using a producer consumer pattern approach. You have a producer who puts the new task description in the queue and you can have multiple consumers (4 or 5 threads) who process from the queue. The number of consumers or the processing thread could vary depending on the load, length of the queue.

Each task involves reading from DB, converting the format, sending to web server and then process the response from web server. I assume each task would do all these steps.

In case of exceptions for an item in the queue, you could potentially mark the queue item as failed and schedule it for a retry later.