Opentracing context not propagated on Quarkus when using reactive messaging

Question

I've two microservices interacting with each other via Kafka, that is the one publishes messages while the other consumes them. Both the publisher and the consumer run on Quarkus (1.12.0.Final) and use reactive messaging and Mutiny.

Producer:

package myproducer;

import myavro.MyAvro;
import io.smallrye.mutiny.Uni;
import org.eclipse.microprofile.reactive.messaging.Channel;
import org.eclipse.microprofile.reactive.messaging.Emitter;
import org.eclipse.microprofile.reactive.messaging.Message;

import javax.enterprise.context.ApplicationScoped;
import javax.inject.Inject;
import java.util.concurrent.CompletableFuture;


@ApplicationScoped
public class Publisher {  
  @Channel("mytopic")
  @Inject
  public Emitter<MyAvro> myTopic;

  @Override
  public Uni<Void> publish(MyModel model) {
    MyAvro avro = MyModelMapper.INSTANCE.modelToAvro(model);

    return Uni.createFrom().emitter(e -> myTopic.send(Message.of(avro)
                                                .addMetadata(toOutgoingKafkaRecordMetadata(avro))
                                                .withAck(() -> {
                                                     e.complete(null);
                                                     return CompletableFuture.completedFuture(null);
                                                })));
  }
}

Consumer:

package myconsumer;

import myavro.MyAvro;
import io.smallrye.mutiny.Uni;
import io.smallrye.reactive.messaging.kafka.IncomingKafkaRecord;
import org.eclipse.microprofile.reactive.messaging.Incoming;

import javax.enterprise.context.ApplicationScoped;

@ApplicationScoped
public class Consumer {

  @Incoming("mytopic")
  public Uni<Void> consume(IncomingKafkaRecord<String, MyAvro> message) {
    MyModel model = MyModelMapper.INSTANCE.avroToModel(message.getPayload());

    return ...;
  }

}

Dependencies: include among others the artefacts

• quarkus-smallrye-reactive-messaging-kafka
• quarkus-resteasy-mutiny
• quarkus-smallrye-opentracing
• quarkus-mutiny
• opentracing-kafka-client

Quarkus configuration (application.properties): includes among others

quarkus.jaeger.service-name=myservice
quarkus.jaeger.sampler-type=const
quarkus.jaeger.sampler-param=1
quarkus.log.console.format=%d{yyyy-MM-dd HH:mm:ss} %-5p traceId=%X{traceId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t) %s%e%n

mp.messaging.incoming.mytopic.topic=abc
mp.messaging.incoming.mytopic.key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
mp.messaging.incoming.mytopic.value.deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
...
mp.messaging.incoming.mytopic.interceptor.classes=io.opentracing.contrib.kafka.TracingConsumerInterceptor

With this setup no traceId or spanId is logged at all (even though they should according to Quarkus' "Using OpenTracing" guide). Only after adding @org.eclipse.microprofile.opentracing.Traced a traceId and a spanId is set, but both are completely unrelated to each other on the producer and the consumer.

I checked my opentracing configuration against the beforementioned Quarkus' guide "Using OpenTracing" but found no hints for a misconfiguration on my side. After reading discussions about issues in some Quarkus extensions relying on ThreadLocals when using with Mutiny I added the artefact quarkus-smallrye-context-propagation to my dependencies, but to no avail.

I suspect that the issue might be related to https://github.com/quarkusio/quarkus/issues/15182 , though there it's about reactive routes instead of reactive messaging.

Any ideas?

Answer 1

This issue is not easy to solve, first I will try to explain what happens.

OpenTracing has the concepts of transactions and spans. A span is a block of execution (a method, a database call, a send to a Kafka topic), whereas a transaction is a distributed process that span multiple components (a group of spans).

The issue here is that, each time a span is created, it didn't find any OpenTracing transaction so it creates a new one. This is why none of your spans are correlated to each others.

In OpenTracing, when you create a span, you'll create it based on a span context. Each OpenTracing integration will creates a span context based on the extension technology (I didn't find a better term), for example, HTTP span context is based on HTTP headers and Kafka span context is based on Kafka Headers.

So, to correlate two spans, you need to have the span context created with some context from the underlying technology providing the right OpenTracing ID .

For example, to correlate two Kafka spans, you need to have a uber-trace-id header (this is the default name of the OpenTracing id in Jaeger) with the trace identifier (see tracespan-identity for the format of this header).

Knowing this, there is multiple things to do.

First, you need to add an uber-trace-id Kafka header inside your outgoing message in your @Traced method to correlate the span from the method with the span created inside the Kafka producer interceptor.

Tracer tracer = GlobalTracer.get(); // you can also inject it
JaegerSpanContext spanCtx = ((JaegerSpan)tracer.activeSpan()).context();
// uber-trace-id format: {trace-id}:{span-id}:{parent-span-id}:{flags}
//see https://www.jaegertracing.io/docs/1.21/client-libraries/#tracespan-identity
var uberTraceId = spanCtx.getTraceId() + ":" +
        Long.toHexString(spanCtx.getSpanId()) + ":" +
        Long.toHexString(spanCtx.getParentId()) + ":" +
        Integer.toHexString(spanCtx.getFlags());
headers.add("uber-trace-id", openTracingId.getBytes());

Then, you need to correlate your @Traced method with the span from the incoming message, if any. For this, the easiest way is to add a CDI interceptor that will try to create a span context for all methods annotated with @Traced based on the method parameters (it will search for a Message parameter). For this to work, this interceptor needs to be executed before the OpenTracing interceptor, and sets the span context in the interceptor context.

This is our interceptor implementation, feel free to use it or adapt it for your needs.

public class KafkaRecordOpenTracingInterceptor {

    @AroundInvoke
    public Object propagateSpanCtx(InvocationContext ctx) throws Exception {
        for (int i = 0 ; i < ctx.getParameters().length ; i++) {
            Object parameter = ctx.getParameters()[i];

            if (parameter instanceof Message) {
                Message message = (Message) parameter;

                Headers headers = message.getMetadata(IncomingKafkaRecordMetadata.class)
                    .map(IncomingKafkaRecordMetadata::getHeaders)
                    .get();
                SpanContext spanContext = getSpanContext(headers);
                ctx.getContextData().put(OpenTracingInterceptor.SPAN_CONTEXT, spanContext);
            }
        }

        return ctx.proceed();
    }

    private SpanContext getSpanContext(Headers headers) {
        return TracingKafkaUtils.extractSpanContext(headers, GlobalTracer.get());
    }
}

This code uses both the Quarkus OpenTracing extension and the Kafka OpenTracing contrib library.

With both the correlation of outgoing message thanks to the addition of the OpenTracing Kafka Header created from the current span context, and the creation of a context from incoming message's header, the correlation should happens in any case.