如何在文件 Scala spark 中输出字段填充？

Question

I have a text file. Now, I want output field padding in file as Exp1 & Exp2 . What should I do? This is my input:

a
a a
a a a
a a a a
a a a a a

Exp1. Fill the remaining fields with the - character when each record in the file does not fit into the n=4 field.

a _ _ _
a a _ _
a a a _
a a a a
a a a a a

Exp2. Same as above. Delete the fields after the n=4 field when the number of fields in the record exceeds n .

a _ _ _
a a _ _
a a a _
a a a a
a a a a

My code:

val df = spark.read.text("data.txt")
val result = df.columns.foldLeft(df){(newdf, colname) => 
   newdf.withColumnRenamed(colname, colname.replace("a", "_"))
} 
result .show 

Answer 1

This resembles a homework-style problem, so I will help guide you based on your provided code and try to lead you on the right path here.

Your current code is only changing the name of the columns. In this case, the column name "value" is being changed to "v_lue". You want to change the actual records themselves.

First, you want to read this data into an RDD. It can be done with a dataframe, but being able to map on the row strings instead of Row objects might make this easier to understand conceptually. I'll get you started.

val data = sc.textFile("data.txt")

Data will be an RDD of strings, where each element is a line in the data file.

We're going to want to map this data to some new data, and transform each row.

data.map(row => {
   // transform each row here
})

Inside this map we make some change to row, which is a string. The code inside applies to every string in the RDD.

You will probably want to split the row to get an array of strings, so that you can count how many occurrences of 'a' there are. Depending on the size of the array, you will want to create a new string and output that from this map. If there are fewer 'a's than n, you will probably want to create a string with enough '_'s. If there are too many, you will probably want to return a string with the correct number.

Hope this helps.