How to create a Dataset of String from a Dataset of List of String Spark Java

Question

I have a Dataset of List of Strings, I need to create anew Dataset from the above having each entry of the list as one line in the new DataSet.

List<String> list = new ArrayList("abc", "def", "ghi");

Dataset<String> input = spark.createDataset(list,Encoders.bean(String.class));

New Dataset is to be like:

"abc
def
ghi"

Answer 1

you can use flatmap to convert the list of String to individual String. // flatMap each line to words in the line

JavaRDD<String> words = input.toJavaRDD().flatMap(s -> Arrays.asList(s.split(",")).iterator());