Connection refused between Azure monitor and Application-insights java agent

Question

Attempting to add Application-insights to a java application running in AKS.

Followed the steps provided here: https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent

• an ai-agent.jar is placed in the root of the image.
  
• The connection string is provided via environment variables.
  
• the image is initialized with the following commands:

command: ["java"]
args: ["-javaagent:/agent.jar", "-jar", "/app.jar"]

both the spring-boot app and the agent do start, but the logs for the agent are followed up by these errors:

2021-08-19 15:43:26.914Z WARN  c.m.a.a.i.q.QuickPulsePingSender - Live metrics endpoint ping failed io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: rt.services.visualstudio.com/157.55.177.6:443 (future failures will be aggregated and logged once every 5 minutes)
reactor.core.Exceptions$ReactiveException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: rt.services.visualstudio.com/157.55.177.6:443
    at reactor.core.Exceptions.propagate(Exceptions.java:392)
    at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:97)
    at reactor.core.publisher.Mono.block(Mono.java:1703)
    at com.microsoft.applicationinsights.agent.internal.quickpulse.QuickPulsePingSender.ping(QuickPulsePingSender.java:95)
    at com.microsoft.applicationinsights.agent.internal.quickpulse.QuickPulseCoordinator.ping(QuickPulseCoordinator.java:106)
    at com.microsoft.applicationinsights.agent.internal.quickpulse.QuickPulseCoordinator.run(QuickPulseCoordinator.java:63)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: java.lang.Exception: #block terminated with an error
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
        ... 5 common frames omitted
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: rt.services.visualstudio.com/157.55.177.6:443
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
    at io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
    at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
    at io.netty.channel.unix.Socket.finishConnect(Socket.java:278)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)
2021-08-19 15:43:27.888Z INFO  c.m.applicationinsights.agent - ApplicationInsights Java Agent 3.2.0-BETA.2 started successfully (PID 1)
2021-08-19 15:43:27.893Z WARN  c.m.a.a.i.i.AppIdSupplier$GetAppIdTask - unable to retrieve appId exception sending request to https://westeurope-3.in.applicationinsights.azure.com/api/profiles/<INSTRUMENTATION_KEY>/appId (future failures will be aggregated and logged once every 5 minutes)
reactor.core.Exceptions$ReactiveException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: westeurope-3.in.applicationinsights.azure.com/13.69.65.23:443
    at reactor.core.Exceptions.propagate(Exceptions.java:392)
    at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:97)
    at reactor.core.publisher.Mono.block(Mono.java:1703)
    at com.microsoft.applicationinsights.agent.internal.init.AppIdSupplier$GetAppIdTask.run(AppIdSupplier.java:123)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: java.lang.Exception: #block terminated with an error
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
        ... 8 common frames omitted
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: westeurope-3.in.applicationinsights.azure.com/13.69.65.23:443
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
    at io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
    at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
    at io.netty.channel.unix.Socket.finishConnect(Socket.java:278)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)

as far as I can tell, the issue seems to stem from the agent not being able to retrieve the appId, but sshing into the container and performing a curl

curl https://westeurope-3.in.applicationinsights.azure.com/api/profiles/<INSTRUMENTATION_KEY>/appId

does return a result, leading me to believe that the pod does have a connection to the Azure monitor instance.

I've tried multiple versions of the agent jar, tried the SDK and custom events inside the spring-boot app, manually starting a secondary java instance, provided the connection string/instrumentation-key using kubectl, local env variable, applicationinsights.json file, and as a JVM parameter, all this to no avail.

right now it feels like I'm all out of ideas and unable to get any further with this issue without some external input. Does anyone recognize this issue, or tackled something similar before?

Answer 1

I've faced similar situation and it seems that in my case, the Java app was running inside a VM without outbound access to the internet. This way,it wasn't reaching Azure Monitor/App Insight DNS and not sending the data to the ingestion service.

Have you tried running a CURL within your app infrastructure and making sure it can outbound http traffic?