Performance counters for the TPL?

Question

Is there any way to see (preferably via built-in performance counter) how much work the TPL has got queued up that it's trying to work through?

Here are some details around what I'm trying to do and what I'm seeing:

I have a large workload that I'm offloading onto the .NET Threadpool using Task.Factory.StartNew() (default scheduler) . When there is more load than my system can handle, the work queues up until either the load decreases or I run out of memory. I do update a custom performance counter (it's a per-second counter) as part of my processing, and I can see that this counter doesn't go above around 27 000 (messages per second) regardless of how many messages I queue up with Task.Factory.StartNew().

My question is more around monitoring than advice on making the system perform better - I can do micro-optimizations, but at the moment there's no visibility in the perf counters that work is building up. Optimisation aside, I'd like to be able to see exactly how much work the TPL has got buffered away, purely so I can gauge the responsiveness of my system. I'm looking to see whether putting a new message in on the input side will result in immediate processing (zero latency) or whether there will be some latency before I see the results from my request.

What happens now is that below 27 000 messages per second, if I stop the input I see the "Messages processed per second" counter drop to zero. Above this breaking point, the counter continues to register 27k per second for a while (depending on how long I've let the backlog build up) until the rest of the load works its way through the system.

Answer 1

I think the easiest thing would be to maintain another perf counter you increment as each task is queued and decrement as each task starts.

The more involved option is to write a custom TaskScheduler that handles publishing all the required events, but again I imagine this may drastically slow things down unless implemented carefully, but thats the price you pay for instrumentation. You then could then switch the scheduler you use only when you need the instrumentation.

Here is an article on writing a custom TaskScheduler : http://msdn.microsoft.com/en-us/library/ee789351.aspx

If you did write this TaskScheduler implementation though I imagine it would be useful for the community, so please publish the results here and maybe put the code up on GitHub :)