Defining a RestTemplateBuilder bean causes traceId propagation across service boundaries to no longer work

Question

The following code demonstrates that by including a RestTemplateBuilder bean into a project, micrometer-tracing will not propagate traceId s correctly across service boundaries.

I am posting the sample code here as my client prohibits me from sharing code via GitHub.

Create two Spring Maven projects, demo-client and demo-server, referencing the code below, then run the two scripts to get the microservices running. Then, send a GET request to http://localhost:2048/tracer-response and you'll see in the demo-client logs that the traceId s match. Next, uncomment the RestTemplateBuilder bean in DemoClientApplication and re-run the scripts. Retest, and you'll see that the traceId s no longer match.

Please advise as to why this configuration no longer works. It previously worked with Spring Boot 2.7.7 and Spring Cloud 2021.0.5. If the intent from the Spring team is for this configuration to no longer be used and this is pointed out in the documentation, then I apologize for missing it. If it isn't pointed out, then in my opinion this should be highlighted as a warning, as it clearly negates the whole purpose behind using distributed tracing.

demo-client code

package com.example.democlient;

import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.client.RestTemplateBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;

@SpringBootApplication
@Slf4j
public class DemoClientApplication {

  public static void main(String[] args) {
    SpringApplication.run(DemoClientApplication.class, args);
  }

  @Bean
  public RestTemplate restTemplate(RestTemplateBuilder builder) {
    return builder.build();
  }

  // Uncomment the bean definition below, and traceId propagation will no longer work, meaning the traceId
  // values logged by demo-client and demo-server will not be the same value.
/*  @Bean
  public RestTemplateBuilder restTemplateBuilder() {
    return new RestTemplateBuilder();
  }*/
}

package com.example.democlient;

import io.micrometer.tracing.Tracer;
import jakarta.annotation.security.PermitAll;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;

@RestController
@Slf4j
public class DemoClientController {

  @Autowired RestTemplate restTemplate;
  @Autowired Tracer tracer;

  @PermitAll
  @GetMapping(value = "/tracer-response", produces = MediaType.APPLICATION_JSON_VALUE)
  public TracerResponse getClientResponse() {
    log.info("Getting the tracer response from demo server...");
    var clientTraceId = tracer.currentSpan().context().traceId();
    var clientSpanId = tracer.currentSpan().context().spanId();
    log.info(
        "Here's the traceId and spanId before the call: {} and {}", clientTraceId, clientSpanId);
    var serverTracerResponse =
        restTemplate
            .getForEntity("http://localhost:2049/tracer-response", TracerResponse.class)
            .getBody();
    var serverTraceId = serverTracerResponse.getTraceId();
    var serverSpanId = serverTracerResponse.getSpanId();
    log.info(
        "Here's the traceId and spanId from the server: {} and {}", serverTraceId, serverSpanId);
    log.info("Do the client and server traceIds match? {}", clientTraceId.equals(serverTraceId));
    return serverTracerResponse;
  }
}

package com.example.democlient;

import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.Data;

@JsonInclude(JsonInclude.Include.NON_NULL)
@Data
public class TracerResponse {
  private String spanId;
  private String traceId;
}

src/main/resources/application.yml

server:
  port: 2048
spring:
  application:
   name: demo-client

src/main/resources/logback-spring.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include
            resource="org/springframework/boot/logging/logback/defaults.xml"/>
    ​
    <springProperty scope="context" name="springAppName"
                    source="spring.application.name"/>
    <property name="LOG_PATTERN"
              value="%d{yyyy-MM-dd, America/New_York} | %d{HH:mm:ss.SSS, America/New_York} | %-16.16thread | %-5.5p | %-40.40logger{40} | %-39.39(Trace: %X{traceId}) | %-22.22(Span: %X{spanId}) | %m%n"/>

    <appender name="console"
              class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>${LOG_PATTERN}</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <logger name="com.example" level="DEBUG" additivity="false">
        <appender-ref ref="console"/>
    </logger>

    <logger name="org.springframework" level="WARN"
            additivity="false">
        <appender-ref ref="console"/>
    </logger>

    <root level="WARN">
        <appender-ref ref="console"/>
    </root>

</configuration>

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
        <parent>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-parent</artifactId>
            <version>3.0.1</version>
            <relativePath/> <!-- lookup parent from repository -->
        </parent>
    <groupId>com.example</groupId>
    <artifactId>demo-client</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>demo-client</name>
    <description>Demo project for Spring Boot</description>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.logging.log4j</groupId>
                    <artifactId>log4j-to-slf4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

demo-server code

package com.example.demoserver;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class DemoServerApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoServerApplication.class, args);
    }

}

package com.example.demoserver;

import io.micrometer.tracing.Tracer;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@Slf4j
public class DemoServerController {

    @Autowired
    Tracer tracer;

    @GetMapping(value = "/tracer-response", produces = MediaType.APPLICATION_JSON_VALUE)
    public TracerResponse getClientResponse() {
        log.info("Getting the serverResponse...");
        log.info("traceId: {} spanId: {}",tracer.currentSpan().context().traceId(),tracer.currentSpan().context().spanId());
        var response = new TracerResponse();
        response.setTraceId(tracer.currentSpan().context().traceId());
        response.setSpanId(tracer.currentSpan().context().spanId());
        return response;
    }
}

package com.example.demoserver;

import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.Data;

@JsonInclude(JsonInclude.Include.NON_NULL)
@Data
public class TracerResponse {
    private String traceId;
    private String spanId;
}

src/main/resources/application.yml

server:
  port: 2049
spring:
  application:
    name: demo-server

src/main/resources/logback-spring.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include
            resource="org/springframework/boot/logging/logback/defaults.xml"/>
    ​
    <springProperty scope="context" name="springAppName"
                    source="spring.application.name"/>
    <property name="LOG_PATTERN"
              value="%d{yyyy-MM-dd, America/New_York} | %d{HH:mm:ss.SSS, America/New_York} | %-16.16thread | %-5.5p | %-40.40logger{40} | %-39.39(Trace: %X{traceId}) | %-22.22(Span: %X{spanId}) | %m%n"/>

    <appender name="console"
              class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>${LOG_PATTERN}</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <logger name="com.example" level="DEBUG" additivity="false">
        <appender-ref ref="console"/>
    </logger>

    <logger name="org.springframework" level="WARN"
            additivity="false">
        <appender-ref ref="console"/>
    </logger>

    <root level="WARN">
        <appender-ref ref="console"/>
    </root>

</configuration>

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.0.1</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>demo-server</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>demo-server</name>
    <description>Demo project for Spring Boot</description>
    <properties>
        <java.version>17</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.logging.log4j</groupId>
                    <artifactId>log4j-to-slf4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-properties-migrator</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

scripts to manage and run the microservices

kill_demo_server.sh

#! /bin/bash
set -e
set -x

ps -ef | grep 'demo-server-0.0.1' | kill -9 $(awk 'NR==1{print $2}') 2>/dev/null;

build_deploy_demos.sh

#! /bin/bash
set -e
set -x

cd ~/repos/demo-server;
mvn clean install;
nohup java -jar target/demo-server-0.0.1-SNAPSHOT.jar &
cd ~/repos/demo-client;
mvn clean install;
java -jar target/demo-client-0.0.1-SNAPSHOT.jar

Answer 1

Not clear by your question why do you need a custom RestTemplateBuilder bean, but I believe there are some use-case anyway.

So, let's see if making that bean fully similar to what Spring Boot auto-configuration does helps to solve your requirements!

See RestTemplateAutoConfiguration :

@Bean
@Lazy
@ConditionalOnMissingBean
public RestTemplateBuilder restTemplateBuilder(RestTemplateBuilderConfigurer restTemplateBuilderConfigurer) {
    RestTemplateBuilder builder = new RestTemplateBuilder();
    return restTemplateBuilderConfigurer.configure(builder);
}

Therefore your custom bean should be similar:

  @Bean
  public RestTemplateBuilder restTemplateBuilder(RestTemplateBuilderConfigurer restTemplateBuilderConfigurer) {
       RestTemplateBuilder builder = new RestTemplateBuilder();
       return restTemplateBuilderConfigurer.configure(builder);
  }

That RestTemplateBuilderConfigurer delegates to respective customizers. At the moment I see this impls in my classpath:

MockServerRestTemplateCustomizer
ObservationRestTemplateCustomizer
TraceRestTemplateCustomizer

Where ObservationRestTemplateCustomizer is the one which instruments a RestTempalte with micrometer-tracing.