在多个服务/组件之间跟踪性能的最佳方法是什么

Question

Our application consists of several services. Let's say:

1. An Authorization Service
2. Business Service Alpha
3. Business Service Beta
4. Etc.

Most of these services are .Net, but not necessarily all of them. Most are under are our control, however there two are external services and there may eventually be more. Eventually, our .Net services will be moved into the cloud.

End-to-end performance is important and right now, we're struggling to see the big picture of where the performance bottlenecks are. A couple of ideas that we're considering are:

1. Creating a common log file that each of our .Net components will write to. We will look for tools that can generate end-to-end waterfalls.
2. Create an asynchronous logging service that will receive logging calls. We're not certain whether this would also write log files our use some other sort of storage.

For external services, we only expect the time called and the time returned, but for those services under our control, we want to track more granular events.

Any course of action will involve a bit of effort. Before we begin, what pattern and/or tools do you think is the best way to provide effective performance data across multiple services.

Answer 1

The main question is: How much is it worth to you.

There are plenty cloud or self host solutions on the marked which will allow you to track performance / errors and much more.

My only suggestion is: Don't reinvent the weel, take a prebuild solution. The costs would propably be lower as if you would do it yourself.

It's hard to answer your question without advertising but since you mentioned .NET

Microsoft offers something called Application Insight: https://azure.microsoft.com/en-us/documentation/articles/app-insights-windows-get-started/

It's as simple as

 var tc = new TelemetryClient(); // Call once per thread

 // Send a user action or goal:
 tc.TrackEvent("Win Game");

 // Send a metric:
 tc.TrackMetric("Queue Length", q.Length);

 // Provide properties by which you can filter events:
 var properties = new Dictionary{"game", game.Name};

 // Provide metrics associated with an event:
 var measurements = new Dictionary{"score", game.score};

 tc.TrackEvent("Win Game", properties, measurements);

Take a look at it, compare some other tools and pick one that fits your needs.

Answer 2

I've concluded that the following do-it-yourself option provides value:

1. Add a GUID that will flow throughout the process.
2. Add some type of toggle that can be used to turn on "verbose" logging on demand, for an individual request.
3. When "verbose" logging has been chosen, create a single log file for that GUID and log the timings to a single event log.

This is not an industrial strength solution, however when the individual components/services are under your control, it enables you to easily, at a glance, gain an understanding of the high level performance flow through the system. This is useful during development.

As we discover which pieces of the puzzle cause the greatest bottlenecks, we will then turn to more specialized tools to analyze performance on a more granular basis.