Spark在具有2个工作程序的群集上的JdbcRDD中引发NullPointerException

Question

I am running spark cluster with 2 Workers, each with 60GB.

I have written below code for JdbcRDD.

String sql   = "SELECT * FROM( SELECT ROW_NUMBER() OVER(ORDER BY (SELECT 1)) AS Row,"+ 
               " * FROM [Table_1]) A WHERE Row >= ? AND Row <= ? ";

    SparkContext sctx   = new SparkContext(getSparkConf());
    try {
        JdbcRDD<List> jdbcRdd = new JdbcRDD(sctx,new GetJDBCConnection(),sql,0, rowCount, 200, new GetJDBCResult(),scala.reflect.ClassTag$.MODULE$.AnyRef());

        Object[] bb = (Object[])jdbcRdd.collect();

        System.out.println("Length of Object array : "+bb.length);
        System.out.println("JdbcRDD:- "+bb);
    } catch(Exception e) {
        e.printStackTrace();
    }

and code for GetJdbcResult is

class GetJDBCResult extends AbstractFunction1<ResultSet, List> implements Serializable{

private static final long serialVersionUID = -78825308090L;
 public List apply(ResultSet rs) {
    Object result = null;

    List lst = new ArrayList();
    try {
        System.out.println("In apply method");
        System.out.println("resultSet : -"+rs);
        int cols  = rs.getMetaData().getColumnCount();
        System.out.println("no of columns : "+cols);
        for(int i=1;i<=cols;i++) {
            result = rs.getObject(i);

            System.out.println("Object : -"+result);
            lst.add(result);
        }
        System.out.println("result->" + lst);
    } catch (SQLException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    return lst;         
}

}

Above code runs fine I run Spark on standalone mode (local*) but If use cluster environment, then it throws below error :

org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 0.0 failed 4 times, most recent failure: Lost task 7.3 in stage 0.0 (TID 39, DD1AI7511): java.lang.NullPointerException: 
    org.apache.spark.rdd.JdbcRDD$$anon$1.<init>(JdbcRDD.scala:74)
    org.apache.spark.rdd.JdbcRDD.compute(JdbcRDD.scala:70)
    org.apache.spark.rdd.JdbcRDD.compute(JdbcRDD.scala:50)
    org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
    org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
    org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
    org.apache.spark.scheduler.Task.run(Task.scala:54)
    org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)

Driver stacktrace:

There are no traces/logs on Worker logs. Am I doing something wrong here ? Anybody has any idea ?

Answer 1

If you look at line 74 in JdbcRDD, it is very clear that this is due to database connection being null.

https://github.com/apache/spark/blob/655699f8b7156e8216431393436368e80626cdb2/core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala

Since it is working in local but not in cluster, check whether your cluster instances have access to database. If this is in EC2, please make sure your firewalls rules are correct. Also make sure spark cluster is running in the same VPC as your database. Best way to verify this is to SSH to one of your spark slave node and see whether you can connect to database remotely from there.