添加jmx__javaagent后无法在kubernetes中列出kafka主题

Question

I have a running kafka pod in my kubernetes cluster. For getting custom metrics in prometheus format, I have configured a jmx_prometheus_javaagent and assagined the port 2255. And I am able to list the metrics from localhost:2255/metrics.

The issue is, after this, I am not able to list the kafka topics. Getting following error:

---

bash-4.3# /opt/kafka/bin/kafka-topics.sh --list --zookeeper dz-zookeeper:2181

Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.net.httpserver.ServerImpl.bind(ServerImpl.java:133)
at sun.net.httpserver.HttpServerImpl.bind(HttpServerImpl.java:54)
at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer.<init>(HTTPServer.java:145)
at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:49)
FATAL ERROR in native method: processing of -javaagent failed

Aborted (core dumped)

---

Any idea, how to solve this error?

Answer 1

You've set it up so that the java agent is being loaded not just for the kafka server, but also all of the command line tools. You should change your configuration so it is only being loaded for the server.

Answer 2

Get the kafka broker container ID by using "docker ps"

Then run the kafka command against that container from the command line this way:

docker exec -it CONTAINERID /bin/bash -c "KAFKA_OPTS= && kafka-topics --zookeeper 127.0.0.1:2181 --list"

Change the CONTAINERID and the zookeeper address to fit your enviroment.