使用Filesystem Watcher和/或MSMQ wcf服务进行多线程

Question

I need to create a service which is basically responsible for the following:

1. Watch a specific folder for any new files created.
2. If yes , read that file , process it and save data in DB.

For the above task, I am thinking of creating a multi threaded service with either of the following approach:

1. In the main thread, create an instance of filesystem watcher and as soon as a new file is created, add that file in the threadQueue. There will be N no. of consumer threads running which should take a file from the queue and process it (ie step 2).

2. Again in the main thread, create an instance of filesystem watcher and as soon as a new file is created, read that file and add the data to MSMQ using wcf MSMQ service. When the message is read by the wcf msmq service, it will be responsible for processing further

I am a newbie when it comes to creating a multi threaded service. So not sure which will tbe the best option. Please guide me.

Thanks,

Answer 1

First off, let me say that you have taken a wise approach to do a single producer - multiple consumer model. This is the best approach in this case.

I would go for option 1, using a ConcurrentQueue data structure, which provides you an easy way to queue tasks in a thread-safe manner. Alternatively, you can simply use the ThreadPool.QueueUserWorkItem method to send work directly to the built-in thread pool, without worrying about managing the workers or the queue explicitly.

Edit : Regarding the reliability of FileSystemWatcher , MSDN says:

The Windows operating system notifies your component of file changes in a buffer created by the FileSystemWatcher. If there are many changes in a short time, the buffer can overflow. This causes the component to lose track of changes in the directory, and it will only provide blanket notification. Increasing the size of the buffer with the InternalBufferSize property is expensive, as it comes from non-paged memory that cannot be swapped out to disk, so keep the buffer as small yet large enough to not miss any file change events. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties so you can filter out unwanted change notifications.

So it depends on how often changes will occur and how much buffer you are allocating.

Answer 2

I would also consider your demands for failure handling and sizes of the files you are sending. Whether you decide for option 1 or 2 will be dependent on specifications.

Option 2 has the avantage that by using MSMQ you have your data persisted in a recoverable way, even if you may need to restart your machine. Option 1 only has your data in memory which might get lost.

On the other hand, option 2 has a disadvantage that the message size of MSMQ is limited to 4 MB per message (explanation in a Microsoft blog here ) and therefore only half of it when working with unicode characters, while the in-memory queues are capaple of much bigger sizes.

[Edit]

Thinking a bit longer, I would prefer option 2 . In your comment, you mention that you want to move files around in the filesystem. This can be very expensive in regards to performance, even worse if you move the files between different partions.

I have used the MSQM in multiple projects at work and am convinced that it would work well for what you want to do. A big advantage here would be that the MSMQ works with transactional communications. That means, that if for some reason a network or electricity or whatever failure occurs, neither your message nor your files get lost.

If any of those happen while you move a file around it could easily get corrupted.

Only thing I have grumbles in my stomach is the file sizes. To work around the message size limitations of 4 MB (see added link above), I would not put the file content into a message. Instead. I would only send an ID or a filepath with it so that the consuming service can find it and read it when needed.

This keeps the message and queue sizes small and avoids using too much bandwith or memory in network and on your serve(s).