How to perform file logging in a multithreaded app

Question

I need a way of extreme fast logging (frame info about highspeed camera). Its just a few numbers that I need to log, and a simple file.log. Eventlogging is to slow for this.

So then I thought, well just create a file stream so i can lock the file for my app. and append to it.

Normally I would use a simple line such as

Filestream fs = new FileStream(@"D:\Log.csv", FileMode.Append, FileAccess.Write, FileShare.ReadWrite);

inside a method.

However since the frames by the camera driver are executed each in a new thread I got a problem here. As I dont want to re-open and close the file each time the log file is written to. (open and close is slow).

I'd like to open the log file once, at the start of my program, and the threads should only perform a write to it, not closing and opening it again and again.

How to achieve this, since this doesn't work :

using System.IO;
FileStream fs = new FileStream(@"D:\Log.txt", FileMode.Append, FileAccess.Write, FileShare.ReadWrite);

    static void Main(string[] args)
    {
         // doing it in main doesn't work either.
         fs = new FileStream(@"D:\Log.txt", FileMode.Append, FileAccess.Write, FileShare.ReadWrite);
       //...
       //.. init camera and start camera lots of code follows but is not related to the question.
     }

     Camera_Thread.FrameArrived (FrameArrivedEventArgs e)
    {   
       byte[] n = MyFilterFunction(e.frame);         
       fs.WriteByte(MyArrayToString(n));
    }

Answer 1

There's a number of ways, but most if not all involve queueing, especially in a multithreaded environment.

You could use MSMQ to queue your logs for processing, you can also use a separate thread to process logs off an in-memory queue.

string logFile = "Log.txt";
this.Queue = new ConcurrentQueue<string>();

var thread = new Thread(() => 
{
    string log;

    while (true)
    {
        while (!this.Queue.IsEmpty)
        {
            if (!this.Queue.TryDequeue(out log)) continue;

            File.AppendAllText(logFile, "\n" + log);
        }

        Thread.Sleep(1000);
    }

});

thread.Start();

This implementation doesn't take into account how to cancel the logging thread, so I'll let you attempt that on your own first. I'll also add that this isn't very reliable, given the choice, I'd actually use MSMQ.