如何使用Scala在Spark中的Row类型的RDD上进行拆分

Question

I have a file json on HDFS, I read it:

var data = sqlContext.read.json("/.....")

This following, it's Schema:

 |-- @timestamp: string (nullable = true)
 |-- beat: struct (nullable = true)
 |    |-- hostname: string (nullable = true)
 |    |-- name: string (nullable = true)
 |    |-- version: string (nullable = true)
 |-- fields: struct (nullable = true)
 |    |-- env: string (nullable = true)
 |    |-- env2: string (nullable = true)
 |    |-- env3: struct (nullable = true)
 |    |    |-- format: string (nullable = true)
 |    |    |-- name: string (nullable = true)
 |    |    |-- version: double (nullable = true)
 |-- input_type: string (nullable = true)
 |-- text: string (nullable = true)

I want do a split of the field text , I tried by:

var myRDD = data.select("text").rdd

var split_myRDD = myRDD.map(ligne => ligne.split("|"))

It does not work: error: value split is not a member of org.apache.spark.sql.Row

Someone can tell me where's the fault ?

Answer 1

You don't need to convert to RDD for that. You can use split function in DF. The code will look like this.

df.select("text")
  .withColumn("text_split", split(col("text"), "\\|"))

You can also use RDD if there is some special need and use "\\\\|" to split the text. I hope it helps.

Answer 2

When you read your JSON, the resulted object is a DataFrame. When you convert a Dataframe to an RDD, you will have an array of [Row]. A Row Class describe a Row from your Dataframe, and have the same schema as your DataFrame. To be able to take an element from a Row you should do this:

myRDD
  .map(row => row.getString(row.fieldIndex("text")).split("|"))