记录在多个线程中运行的.NET应用程序的最佳实践

Question

We have an application that logs to files currently using streamwriter. As we are looking at scaling up the application, the logging block is coming in the way. The application is multi-threaded and can perform with a thread-count which is in thousands. The thread-count does not hit the CPU as most of the responsibilities is to handle TCP sockets on the fly. But the logging is an issue as there is convergence here and performance of the individual threads is impacted for its TAT here.

I shall like to get input from all of you regarding what kind of architecture/design can be followed for the loggging.

I am thinking of using database as the logging sink. But the writer will have to have a locking mechanism I feel. I don't think, I should try creating large number of loggers in threads. Another option I am thinking of doing a pooling on top of the logger.

Should I also look at MSMQ kind of technologies to quickly move out the traffic?

Answer 1

I would look at existing logging frameworks, such as log4net or Microsoft's Enterprise Library Logging Application Block . (I've used log4net a bit, but not the Logging Application Block.)

I very much doubt that your situation is unique, after all - and if you go for a third-party solution instead of a home-grown one, you'll be able to benefit from the experience of other developers.

If you do roll your own and want a producer/consumer model, you may want to look at the concurrent collections in .NET 4. As mhenrixon mentions though, you'll almost certainly have to find a balance between performance and reliability. If this is just for debug logs then it probably doesn't matter too much if you lose a few if the application shuts down abruptly - but obviously it's a different matter if the logs contain audit records which have to be persisted.

Answer 2

You should also consider NLog , particularly NLog 2.0 (currently in Beta - since September). Among other things, NLog 2.0 adds the ability to log asynchronously . In essence, you would configure the Target(s) (equivalent to log4net Appender) to which you would like to log (file, database, etc). Then, also via configuration, you "wrap" each Target with an asynchronous target wrapper. NLog also has an option in the config file to make all logging asynchronous.

While we you are on the subject of logging, you might also consider using a logging abstraction like Common.Logging for .NET . This gives you the option to defer the final decision on your underlying framework (log4net, NLog, etc) and also gives you the option to write your own logger (which can easily be exposed via Common.Logging).

Answer 3

It sounds like you are after something that in log4net is called Appender Buffering thought that one is not fail safe if logging cannot be "lossy" I recommend using queues. I believe that any of the logging frameworks could easily be extended with MSMQ logging.