从Stringbuilder到RDD

Question

I have a string builder(sb) with data as below in Scala IDE

CellId,Date,Time,MeasType,MeasResult

251498240,2016-12-02,20:45:00,RRC.ConnEstabAtt.emergency,0

251498240,2016-12-02,20:45:00,RRC.ConnEstabAtt.highPriorityAccess,0

251498240,2016-12-02,20:45:00,RRC.ConnEstabAtt.mt-Access,4

Now I want to convert this string into RDD by using scala . Please help me.

I am using this code. But no luck. Thanks in advance

 val headerFile = sc.parallelize(sb)
 headerFile.collect()

Answer 1

StringBuilder is used to build strings from mutable sequence of characters . So what ever you add to the builder would be appended to become as one string.

You would need to separate the strings added to be used as list of strings in sparkcontext

Assuming that the string are added with trailing line feed, you can split the string builder with line feed and use it to be transformed as rdd

val headerFile = sc.parallelize(sb.toString.split("\n"))
headerFile.collect()

To visualize the data, you would have to print them or save them to file

Now if you want to convert to dataframe before saving then you can perform as below

val data = sb.toString.split("\n")
import org.apache.spark.sql.types._
val schema = StructType(data.head.split(",").map(StructField(_, StringType, true)))
val rdd = sc.parallelize(sb.toString.split("\n").tail.map(line => Row.fromSeq(line.split(","))))
spark.createDataFrame(rdd, schema).show(false)

which should give you

+---------+----------+--------+-----------------------------------+----------+
|CellId   |Date      |Time    |MeasType                           |MeasResult|
+---------+----------+--------+-----------------------------------+----------+
|251498240|2016-12-02|20:45:00|RRC.ConnEstabAtt.emergency         |0         |
|251498240|2016-12-02|20:45:00|RRC.ConnEstabAtt.highPriorityAccess|0         |
|251498240|2016-12-02|20:45:00|RRC.ConnEstabAtt.mt-Access         |4         |
+---------+----------+--------+-----------------------------------+----------+