使用 Spark 从大型 Cassandra 表读取错误，获取“远程 RPC 客户端已解除关联”

Question

I set stand alone spark cluster (with cassandra) and i did it but when i read data i get error.My cluster has 3 nodes and each node has 64 GB ram and 20 cores. I'm sharing some Spark-env.sh configuration like spark_executor_cores: 5, spark_executor_memory:5G, spark_worker_cores:20 and spark_worker_memory:45g.

I want to give another information, when i read small table there is no problem but when i read big table i get error. Error description at below. Also when i start pyspark i use this command:

$ ./pyspark --master spark://10.0.0.100:7077
    --packages com.datastax.spark:spark-cassandra-connector_2.12:3.1.0
    --conf spark.driver.extraJavaOptions=-Xss1024m
    --conf spark.driver.port:36605
    --conf spark.driver.blockManager.port=42365

Thanks for your interest

ERROR TaskSchedulerImpl: Lost executor 5 on 10.0.0.10: Remote RPC client disassociated. likely due to containers exceeding threshold, or network issues. Chec driver logs for WARN messages
WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0) (10.0.0.10 executor 5): ExecutorLostFailure (executor 5 exited caused by one of the runnning task) reason: remote RPC client disassociated.
WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1) (10.0.0.11 executor 2):Java.lang.StackOverflowError
 at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1094)
 at java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:184)
 at org.apache.spark.util.ByteBufferInputStream.read(ObjectInputStream.scala:49)
 at java.base/java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2887)
 at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2903)
 at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3678) 
 at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3678)
at java.base/java.io.ObjectInputStream.readString(ObjectInputStream.java:2058)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1663)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2490)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2384)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2222)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1681)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2490)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2384)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2222)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1681)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2490)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2384)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2222)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1681)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2490)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2384)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2222)

Answer 1

The problem you're running into is most likely a networking issue.

It's highly unusual that you need to pin the driver ports with:

    --conf spark.driver.port:36605
    --conf spark.driver.blockManager.port=42365

You'll need to provide background information on why you're doing this.

Also as I previously advised you on another question last week, you need to provide the minimal code + minimal configuration that replicates the problem. Otherwise, there isn't enough information for others to be able to help you. Cheers!