Monitoring .NET ASP.NET Applications

Question

I have a number of applications running on top of ASP.NET I want to monitor. The main things I care about are:

• Exceptions: We currently some custom code which will email us when an exception occurs. If the application is failing hard it will crash our outlook... I know (and use) elmah which partly solves the problem however it is still just a big table of exceptions with a pretty(ish) UI. I want something that makes sense of all of these exceptions (eg groups exceptions, alerts when new ones occur, tells me what the common ones are that I should fix, etc)

• Logging: We currently log to files which are then accessible via a shared folder which dev's grep & tail. Does anyone know of better ways of presenting this information. In an ideal world I want to associate it with exceptions.

• Performance: Request times, memory usage, cpu, etc. whatever stats I can get

I'm guessing this is probably going to be solved by a number of tools, has anyone got any suggestions?

Answer 1

Well, we have exactly the same current solution. Emails upon emails litter my inbox and mostly get ignored. Over the holidays an error caused everyone in dev to hit their inbox limit ;)

So where are we headed with a solution?

1. We already generate classes for all our excpetions, they rarely get used from more than one place. This was essentially step one, but we started the code base this way.
2. We modified the generator to create unique HRESULT values for all exceptions.
3. Again we added a generator to create a .MC message resource file for every exception.
4. Now every exception can write itself to the Windows Event Log, and thus we removed all emails etc, and rely on the event log.
5. Now with an event log full of information, including unique message codes and parameters for each exception, we can use off-the-shelf tools to aggregate, monitor, and alert.

The exception generator (before modifications above) is located here:

• http://csharptest.net/browse/src/Tools/Generators

It integrates with visual studio by replacing the ResX generator with this:

• http://csharptest.net/browse/src/Tools/CmdTool

I have not yet published the MC generator, or the HRESULT generation; however, it will be available in the above locations when I get the time.

-- UPDATE --

All the tools and source are now available online for this. So where do I go from here?

1. Download the source or binaries from: http://code.google.com/p/csharptest-net/
2. Take a look at the help for CmdTool.exe Visual Studio Integration
3. Then review the help on Generators for ResX and MC files , there are several ways to generate MC files or complete message DLLs from ResX files. Pick the approach that fits you best.
4. Run CmdTool.exe REGISTER to register the tool with Visual Studio
5. Create a ResX file as normal, then change the custom tool to CmdTool
6. You will need to add some entries to the resx file. At minimal create the following:
   • ".AutoLog" = true
   • ".NextMessageId" = 1
   • ".EventSource" = "HelloWorld"
   • "MyCustomException" = "Some exception text"
   • Other settings exampled by the NUnit: http://csharptest.net/browse/src/Tools/Generators/Test/TestResXAutoLog.cs#80
7. Now you should have an exception class being generated that writes log events. You will need to build the MC file as a pre/post build action with something like:
   • CSharpTest.Net.Generators.exe RESXTOMESSAGEDLL /output=MyCustomMessages.dll /input=TheProjectWithAResX.csproj
8. Lastly, you need to register it, run the framework's InstallUtil.exe MyCustomMessages.dll

That should get you started until I get time to document it all.

Answer 2

You should take a look at Gibraltar not used it myself but looks very good! Also works with nLog and log4net so if you use those you are in luck!!

Answer 3

In terms of handled exceptions or just typical logging l4ndash is worth a look. I always set our log4net to not only write out text files, but to append to the database. That way l4ndash can analyse it easily. It'll group your errors, let you see where bad things are occurring a lot. You get one free dev license

With Elmah we just pull down the logs periodically. It can exports as csv, then we use Excel do filter/group the data. It's not ideal, but it works. It would be nice to write a script to automate this a bit more. I've not seen much else out there for Elmah.

Answer 4

一个建议从Ryans罗伯茨我真正喜欢的是exceptioneer这似乎至少要解决我的例外困境。

Answer 5

I would first go for log4net . The SmtpAppender can wait for N exceptions to cumulate before sending an email and avoid crashing Outlook. And log4net also logs to log files that can be stored on network drives, read with cat and grep, etc.

About stats, you can perform a health/performance logging with the same tools, ie. spawn a thread that every minute logs CPU usage etc.

I don't have a concrete answer for the first part of question, since it implies automated log analysis and coalescence . At university, we made a tool that is designed to do part of these things but doesn't apply to your scenario (but it's two-way integrated with log4net).

Answer 6

您可以通过IIS日志运行LogParser来获取请求时间（以及其他任何已保存的）的度量标准。

Answer 7

We have built a simple monitoring app that sits on the desktop and flashes up red when there is either an exception written to the event log from one of the apps on the server or it writes an error to the error log table on the database. It also monitors the database health, checking fragmentation and the like.

The only problem we have with this is that it can be a little intrusive on the desktop as it keeps popping up with a red message box if there is aa problem. However it does encourage you to fix it asap.

We currently have this running on several of the developers machines. The improvement we are thinking of making is to have one monitoring app running on a server that then publishes an rss feed so that the app is only checking once in one place but we can consume the information from anywhere using whichever method we choose at the time (such as through our phones when we aren't in the office).

Answer 8

You can have an RSS feed select from your Exceptions table (and other things). Then you can subscribe to the RSS feed in MS Outlook or on any smart phone. I use an RSS feed reader called NewsRob because it alerts me when there is something new.

I blog about how to do this HERE.

As a related step, I found a way to notify myself when something DIDN'T happen. That blog is HERE.