What are the best practices for including logging using log4net?

Question

I have been told to add “logging” to my code using log4net, the problem is no one can travel in time and see what real world problems the logging will need to be used to solve.

Therefore is there a set of guidlines anyway as to what to log so as to get a reasonable cost/benefit trade off ?

Therefore:

What kinds of logging should one add to an application that would be useful later?

(The code uses a lot of WCF , one side is in Winforms, the other side is a “server” that normally run on the same machine)

--

I have excepted AJM's answers to do the useful blog post with lot of comments it points to, but if someone comes up with a nice set of " rules of thumb " I am likely to change the expected answer.

Answer 1

One thing to bear in mind is that whilst your configuration will handle logging at different levels, you may be causing heavy overhead in your log calls. For example:

// some kind of loop
// do some operations
Logger.LogDebug(myObject.GetXmlRepresentation());
// end loop

This will obviously only log the object if you have a logger listening to DEBUG logs, however the call to build the XML object will run regardless of your logging level and could cause some considerable slowdowns.

---

The correct solution would be:

// some kind of loop
// do some operations
if (Logger.IsDebug)
{
    Logger.LogDebug(myObject.GetXmlRepresentation());
}
// end loop

Answer 2

I found this article very helpful: http://blog.codinghorror.com/the-problem-with-logging/

In particular I think a minimalist approach is indeed the way to go. In the past I've tried to log too much but this bloats the code

Also the thinking that the more log entries the better is wrong because it bloats the logs themselves. I now look at loggings main benefit as providing a "foothold" or overview of what is going on. If more detail is needed for particular areas then so be it but the default position should be less is better

Answer 3

My favorite source of information for this kind of question is Release It - a book from the Pragmatic guys. Highly highly recommended.

Their basic point in regards to your question is that logging should be geared toward what is needed at the operational level. Operations guys are most concerned with exceptional things where the site may be going down (ie connection pool is full, connection to a server is down, etc.) Make sure the messages are self-explanatory and exceedingly clear as to what the problem is, and if applicable what the fix is. Write the messages for human consumption.

I see little point in function entry/exit style logs. Stack traces for top-level caught exceptions are useful, logging around areas where system crash can happen (ie full connection pool) is useful, as is logging around areas where the system crashed before.

Answer 4

In general with logging I add logging in the following order:

1. Function Enter/Exit
2. Major logic steps inside of a function
3. Verbose logging for all intermediate calculations

Obviously I rarely get to the last one, the first one is trivial to do if you roll your own wrapper for log4net and use the disposing pattern, and possibly a little reflection magic.

The 2nd one is done generally during acceptance/integration and regression testing as major logic flow and problem areas are identified. Adding logging at this stage is also fairly minimal as you know in general where you need to add it as you are debugging and testing.

The 3rd is generally (for me anyways) only done in sections of code that experience regressions, or are particularly important.

I have implemented a basic wrapper object for log4net which provides me direct logging capabilities as well as a context object which can be used with IDisposable to wrap "enter/exit" logic in a nice convient package.

Answer 5

One of the great things about log4net is that you can log events to different categories. The defaults are Debug, Info, Warning and Error. I like these to mean

Debug - very verbose, contains lots of debugging information. For example, SQL queries.
Info - useful information that's good to know.
Warning - nothing fatal but an operator should be aware of the problem.
Error - the application is now unstable, the log contains diagnostic information such as the exception message and stack trace.

Use these in code, so eg

_log.Info("Updating object.");

would write an INFO level message to any listener that was interested.

Then you can hook up listeners in configuration to do things with the log messages. Here's one that I'm using:

<log4net>
<appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">
  <layout type="log4net.Layout.PatternLayout">
    <conversionPattern value="%date [%thread] %-5level %logger [%ndc] - %message%newline" />
  </layout>
</appender>
<appender name="FileAppender" type="log4net.Appender.FileAppender">
  <file value="c:\temp\servicelog.txt" />
  <appendToFile value="true" />
  <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
  <layout type="log4net.Layout.PatternLayout">
    <conversionPattern value="%date [%thread] %-5level %logger [%ndc] - %message%newline%exception" />
  </layout>
</appender>
<root>
  <level value="ERROR" />
  <appender-ref ref="ConsoleAppender" />
</root>
<logger name="Warehouse.Core">
  <level value="INFO" />
  <appender-ref ref="FileAppender" />
</logger>
</log4net>

This says: all ERROR messages to the console, all INFO messages from the logger Warehouse.Core to the given file.

Because this wiring of categories to listeners is done in configuration, you can alter the logging after deployment. There's virtually no performance penalty to logging if nothing's listening.

---

Regarding the costs versus benefits of logging there definitely is a sweet spot between too much logging (huge logs that nobody will use) and not enough (a single line that says "fail").

My policy is to log at INFO what could fail: external dependencies (application startup, service calls, SQL connections), and at DEBUG the more complex bits of meaty code (diagnostic messages in business logic, individual SQL calls, some method invocations).

Unusual situations where (for example) defaults are taken that aren't usually would go to WARN, and exceptions go to ERROR or FATAL.

Also: bear in mind that WCF has a most excellent service trace viewer that allows you to "drill down" to individual packets and how they're processed by both ends of the stack. That, too, is available by configuration without code changes. Because of this I'll generally just do very abbreviated logging of WCF service calls and responses.

Answer 6

What kinds of logging should one add to an application that would be useful later?

If you throw exceptions from your own exception classes, or even better, all your exception classes derive from a base class, add ERROR level logging in the (base) exception constructor; saves you having to remember on each catch/throw. Useful if you have a large code base.

For CLR or 3rd party exceptions, log Exception.ToString() not just the message, otherwise you miss the full stack trace (assuming programmers are not swallowing exceptions or re-throwing sans inner exception)

Focus on DEBUG detail in areas where you either know or suspect you will have issues (just ask QA or Tech Support where to look;-)

If you follow the robustness principle , then you may want INFO or WARN logging when you ignore or change inputs, or expected behaviour. This could be useful if your WCF service starts receiving unexpected (but parseable) input.

To ensure your application performs well, don't ship/install it with DEBUG level logging enabled by default, ERROR or WARN is probably the way to go.

I disagree with the last part of the accepted answer, because log4net has awesome filtering capabilities; on the condition that your programmers understand that logging comes at a cost (as per CK's answer) and they (and QA and Tech Support) know about the filters, and that configuring everything at DEBUG is a bad idea. If you are logging at DEBUG level a large object graph, xml document, db result, et cetera, wrap it in some code that reduces the expense:

if (log.IsDebugEnabled)
{
    log.DebugFormat("Loaded in {0} ms {1}", timer.ElapsedMilliseconds, dataSet.GetXml());
}

I would suggest you follow the recommended static logger-per-class approach, simply because it should make logging more useful when you have to actually use it and narrow down on a problem using filters, eg LoggerMatchFilter.

If you follow this approach and are willing take a (fairly small) performance hit, here's one way which uses a stack trace to create ILog objects for any class and ensure the config file is wired up to monitor for changes:

public static class LogFactory
{
    /// <summary>
    /// Create log whose name is the calling methods class name.
    /// </summary>
    /// <remarks>
    /// <para>
    /// Configures the log repository if it hasn't been configured before.
    /// </para>
    /// <para>
    /// Creates a debug log message right after getting the logger, this follows
    /// the log4net recommendation to log first message as early as possible.
    /// </para>
    /// </remarks>
    /// <returns>Log ready for work.</returns>
    public static log4net.ILog Create()
    {
        var method = new StackTrace().GetFrame(1).GetMethod();
        var log = log4net.LogManager.GetLogger(method.DeclaringType);

        if (log4net.LogManager.GetRepository().Configured == false)
        {
            try
            {
                new FileIOPermission(FileIOPermissionAccess.Read,
                    AppDomain.CurrentDomain.SetupInformation.ConfigurationFile)
                    .Demand();

                var configFile = new FileInfo(AppDomain.CurrentDomain.SetupInformation.ConfigurationFile);
                log4net.Config.XmlConfigurator.ConfigureAndWatch(configFile);
                log.DebugFormat("Log4net configured and watching {0}", configFile.FullName);
            }
            catch (System.Security.SecurityException e)
            {
                log.DebugFormat("Unable to watch config file due to security permissions. {0}", e.ToString());
            }
        }

        log.DebugFormat("Logging {0}", log.Logger.Name);

        return log;
    }
}

Answer 7

Logging is not an easy task, but my experience is that all logs should be searchable for responsible parties. An useful target for errors are email directly (and in some cases sms). But in the end all logging data should be searchable in a database with a sesible user interface.

When the email is received to a specific account, this can be processed and put in the database directly. There are som categories and handling rules below:

• Critical errors: should be availbale immediatly email/sms
• Future issues: daily/weekly email
• Debug info ==> daily/weekly email with a notification on how much debug info is produces since last time.

The content of the debugging can be written to the database in different way, but we need to concider performance. Large amount of data should not be written to database in "production mode". This should be done in daily/weekly bases. The best way is to produce a local file (such as XML or plain text) and put this file into the database during maintenance hours (at night). It should be possible to start/stop a debug session and only write the debug information to the database when the debug session is completed.

The debug component could be implemented as a WCF and log2net and directy access to the database and/or a local file store that is put into the database on regular basis.

One thing is clear...ALL errors/exception should be logged somewhere. Nothing is more irritating than lost error messages :)

Happy debugging!

Answer 8

Log events to event viewer.

Use wisely INFO DEBUG WARNING and ERROR.

when developing show everything (use the console on the server side- see below) on production , log only errors (configurable)

I create a logger in the begging of each class giving it the typeof(myClass) but that's optional...

I'm hosting the WCF in DEV as a console app - so watching the server log is easy also in the console, but once it becomes a service, you have to use event viewer....

ah - and it is very fast if you compare it to WCF "calling times" (calling from client to server) so it doesn't really effect your times, unless you has something the logs like crazy (nHibernate for example) )

Answer 9

There's probably a better way to do this for WCF; for WinForms, you might consider looking at PostSharp . This would allow you to log method calls without cluttering your code. Behind the scenes you'd still be using the excellent log4net.

Caveat: I haven't used it myself; I've seen a very impressive presentation on it at a code camp.