以编程方式推断架构以准备Spark数据框数据集 <Row> 来自RDD <Row> 当某些Row对象可能包含不同数量的元素时

Question

I am fetching neo4j node information in spark rdd using neo4j-spark connector . I can obtain RDD<Row> by calling loadNodeRdds() method . But when I try obtaining dataframe calling loadDataframe() method , it throws exception (skip stack trace if you find it too long as main question might turn out to be different in the end):

java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: java.util.Collections$UnmodifiableMap is not a valid external type for schema of string
if (assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object), 0, Condition), StringType), true) AS Condition#4
+- if (assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object), 0, Condition), StringType), true)
   :- assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object).isNullAt
   :  :- assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object)
   :  :  +- input[0, org.apache.spark.sql.Row, true]
   :  +- 0
   :- null
   +- staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object), 0, Condition), StringType), true)
      +- validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object), 0, Condition), StringType)
         +- getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object), 0, Condition)
            +- assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object)
               +- input[0, org.apache.spark.sql.Row, true]

(skipped a lot of rows as it made question reach its character limit)

I was not able to get much from big stackrace above.

So I took JavaRDD<Row> and tried converting it to DataFrame<Row> by programmatically specifying StructType schema .

StructType schema = loadSchema();
Dataset<Row> df = ss.createDataFrame(neo4jJavaRdd , schema);

This threw somewhat similar exception.

So what I did is that I took individual properties of single neo4j node, prepared Row and then JavaRDD<Row> from it and then tried to create dataframe from it by programatically specifying schema as follows:

Row row1 = RowFactory.create("val1", " val2", "val3", "val4", "val5", "val6", 152214d, "val7", 152206d, 11160d, "val8");
List<StructField> fields = new ArrayList<StructField>();
fields.add(DataTypes.createStructField("attr1", DataTypes.StringType, true));
fields.add(DataTypes.createStructField("attr2", DataTypes.StringType, true));
fields.add(DataTypes.createStructField("attr3", DataTypes.StringType, true));
fields.add(DataTypes.createStructField("attr4", DataTypes.StringType, true));
fields.add(DataTypes.createStructField("attr5", DataTypes.StringType, true));
fields.add(DataTypes.createStructField("attr6", DataTypes.StringType, true));
fields.add(DataTypes.createStructField("attrd1", DataTypes.DoubleType, true));
fields.add(DataTypes.createStructField("attr7", DataTypes.StringType, true));
fields.add(DataTypes.createStructField("attrd2", DataTypes.DoubleType, true));
fields.add(DataTypes.createStructField("attrd3", DataTypes.DoubleType, true));
fields.add(DataTypes.createStructField("attr8", DataTypes.StringType, true));

This worked.

So I checked all nodes and realized that not all nodes (that is all Row s in JavaRDD<Row> ) have same number of attributes. This is must be causing data frame preparation to fail. Can I handle it some way programatically without requiring to create and specify pojo .

Answer 1

If you would like to get this done using RDDs as you mentioned, do as follows:

• Before trying to transform the (RDD + schema) to a dataframe, go over the RDD (using the map function), and make sure each row has all the relevant attributes.
• If an attribute is not present in a row, add it and make it null.

After that, your RDD rows will have the same schema, and the transformation to a dataframe will work.

Answer 2

There are certain things that I came to realize while working with neo4j-spark-connector that I want to share here.

1. In general, if you are going to prepare dataframe , its not preferrable to return neo4j's object types, specifically node and relationship. That is something like below returning node is not preferrable:

    MATCH(n {id:'xyz'}) RETURN n 

   Instead return properties:

    MATCH(n {id:'xyz'}) RETURN properties(n) 

2. If you are unsure that all nodes will not have same number of properties , then its better to return them explicitly, instead of returning properties and obtaining JavaRDD. Since that will require us to process JavaRDD again to add NULL for non existent properties. That, is instead of doing this:

    MATCH(n {id:'xyz'}) RETURN properties(n) 

   return in this way:

    MATCH(n {id:'xyz'}) RETURN n.prop1 AS prop1, n.prop2 AS prop2, ..., n.propN AS propN 

   Neo4j will itself add NULL s for non existing properties as it can be seen image below, we dont have to iterate over them again. By returning this, I was able to obtain neo4j node information directly by using loadDataframe() method.