Apache Flink throws UnknownHostException on cluster

Question

I have a flink project that is connecting to nifi to pull data. The setup to pull get the datastream works just fine when running locally.

.url("http://1.2.3.4:8080/nifi")
            .portName("MyPortName")
            .requestBatchCount(5)
            .buildConfig();

But when I add the .jar to the remote cluster and run the job it throws this:

java.net.UnknownHostException
at sun.nio.ch.Net.translateException(Net.java:177)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:127)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:712)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:685)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:301)
at org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:129)
at org.apache.flink.streaming.connectors.nifi.NiFiSource.run(NiFiSource.java:90)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:78)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:56)
at org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:39)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
at java.lang.Thread.run(Thread.java:745)

The only reason I can find for an UnknownHostException is that it is because the IP of the host name can't be resolved, but I am giving the IP already. There was an issue earlier with it being unable to connect to nifi because I have to set what IP is allowed to access the nifi instance. So I added the AWS server as allowed and it fixed that, but obviously I have this now.

Any help is greatly appreciated!

Answer 1

I figured the problem. I had my nifi cluster and my flink cluster in different regions. Moved the flink cluster to the same region and used either the public or private url for the cluster and it works fine.