Ways to trace calls between services

Question

I'm developing several services on Java that communicate with each other through HTTP. Logback is used as logging framework.

In case of error within one service it's not difficult to define error reason looking through the service logs. But if we have a downstream call that affects a number of services:

Service A -> Service B -> Service C -> Service D

and the last service failed with an error, I need some means to trace the call up to Service A to troubleshoot the issue.

Are there any ready solutions for the problem? Should I add some additional attribute to log message that is unique for each downstream call?

Answer 1

If you want to trace the flow of execution you need some sort of tag on each message. You will need:

• a request identifier
• a service identifier.

When a flow is started you tag the first call with the request identifier and the service identifier. On each service that handles the call and needs to make other calls, to other services, you keep the request identifier and add the current service identifier to the one of the request you received.

Basically the request identifier allows you to identify a flow of calls, while the service identifiers allow you to see the hops in the flow.

For your example, Service A -> Service B -> Service C -> Service D , you might have something like this:

1. Service A starts a call "123-A";
2. service B receives this and needs to make a call to service C, so it uses "123-A" and ads his identifier: "123-A,B";
3. service C receives this and needs to make a call to service D, so it uses "123-A,B" and ads his identifier: "123-A,B,C";
4. if something happens with the request in service D, you know on what service logs to look (A,B,C) and what to look for (123).

You can have these in the HTTP message or as HTTP headers, the idea is to somehow add them to the log message whenever you log something.

Of course the above is for a simple case as the one you gave an example of, so it can be extended for other cases. The idea is to treat the calls as a sort of stack. On each hop you "push" some identification into the message.

Answer 2

There is actually a solution for spring-based services: request correlation spring cloud starter .

The filter adds unique request identifier as a header (if there is no such header) to each request. One can put the id to MDC and add it to log entries with service id.