Using ByteBuddy to instrument another java agent

Question

I'm using the Elastic APM Agent as a Java Agent to monitor usages of various methods in my spring boot microservice. This all works fine and we're able to graph various metrics in Kibana. Unfortunately what it doesn't do is consistently attach the same labels to all spans within a transaction eg details of the user who made the original request.

To work around this I thought I could use ByteBuddy (which I've never used before) to wrap any usages of the APM Span class and attach that information (as it's readily available from ThreadLocal) to each instance. I am however having issues accessing the Span class as it's in the APM Java Agent and with the following code I get the following logs where it appears that it's not able to find the Span class...

        Instrumentation instrumentation = ByteBuddyAgent.install();
        new AgentBuilder.Default()
                .with(debuggingListener)
                .ignore(ElementMatchers.nameStartsWith("net.bytebuddy."))
                .type(ElementMatchers.named("co.elastic.apm.agent.impl.transaction.Span"), ElementMatchers.isBootstrapClassLoader())
                .transform((builder, type, classLoader, module) -> builder.visit(Advice.to(SpanAdvice.class).on(ElementMatchers.hasMethodName("start"))))
                .installOn(instrumentation);

[Byte Buddy] DISCOVERY co.elastic.apm.agent.impl.transaction.Transaction [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] IGNORE co.elastic.apm.agent.impl.transaction.Transaction [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] COMPLETE co.elastic.apm.agent.impl.transaction.Transaction [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] DISCOVERY co.elastic.apm.agent.impl.transaction.Transaction$1 [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] IGNORE co.elastic.apm.agent.impl.transaction.Transaction$1 [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] COMPLETE co.elastic.apm.agent.impl.transaction.Transaction$1 [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] DISCOVERY co.elastic.apm.agent.impl.transaction.AbstractSpan$ChildDurationTimer [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] IGNORE co.elastic.apm.agent.impl.transaction.AbstractSpan$ChildDurationTimer [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] COMPLETE co.elastic.apm.agent.impl.transaction.AbstractSpan$ChildDurationTimer [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] DISCOVERY co.elastic.apm.agent.impl.transaction.SpanCount [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] IGNORE co.elastic.apm.agent.impl.transaction.SpanCount [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] COMPLETE co.elastic.apm.agent.impl.transaction.SpanCount [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] DISCOVERY co.elastic.apm.agent.impl.transaction.TraceState$TextTracestateAppender [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] IGNORE co.elastic.apm.agent.impl.transaction.TraceState$TextTracestateAppender [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] COMPLETE co.elastic.apm.agent.impl.transaction.TraceState$TextTracestateAppender [null, unnamed module @758a34ce, loaded=false]

I've tried using ByteBuddy for my own classes and it all works without issue, but I'm getting very confused around which classloader has loaded what and how to point ByteBuddy at them.

Answer 1

If the objective is to attach labels to a transaction over multiple spans then using the public APIs from the Elastic APM for Java is a better choice instead of instrumenting the JVM with ByteBuddy. You will have much more freedom to do what you want to do without relying on a hacking. FYI, the Elastic APM agent for Java already instrument the JVM with additional bytecode so what you are doing may get even more confusing because of this.

Alternatively, you can also use the OpenTracing Bridge to set labels in a transaction.

Answer 2

Disclaimer: This answer is a stub for now.

You should first try to explore a more canonical way of doing things like Ricardo suggested. If for some reason that does not work, then we could explore ways to instrument your agent class - not so much because I think it is a good idea but because it is technically interesting.

Basically, we would have to find out if maybe the class you want to instrument was already loaded before your ByteBuddy agent gets active. Then you would have to use class retransformation rather than redefinition. You would have to make sure the advice you apply can do its job without the need to change the class structure with regard to method signatures and fields.

  .disableClassFormatChanges()
  .with(RETRANSFORMATION)

You would also need to make sure that the advice and ByteBuddy are visible to the other agent's classloader, eg by putting both on the boot class path. But let's not get ahead of ourselves. Explore Ricardo's ideas first, please.