apache spark - 無法連接到遠程spark獨立系統

Question

我的獨立spark正在ec2實例上運行。 但是當我嘗試通過Java代碼連接到它時，我收到了一個錯誤。 Java代碼是：

    SparkConf conf = new SparkConf().setAppName("App_Name")
            .setMaster("spark://172.31.2.165:4040");
            //.set("spark.ui.port","4040");
    JavaSparkContext context = new JavaSparkContext(conf);

錯誤是：

java.io.IOException: Failed to connect to /172.31.2.165:4040
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
    at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:200)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:183)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection timed out: no further information: /172.31.2.165:4040
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
    ... 1 more

Answer 1

• 默認情況下，您應該使用端口7077連接到主服務器，端口4040用於Web UI
• 確保端口7077可以從您運行Spark Business的任何地方訪問，例如在從EC2外部運行主服務器時在主服務器所在的安全組中打開它