如何在spark scala中写一个固定宽度的output文件

Question

Hi Now out of a spark Scala data frame I am getting a text file output with comma separated values and its coming in a folder as part files. I wanted it to as fixed width like first column should be 10 bytes, next should be 5 bytes, 3rd should be 8 bytes..so on and as a single output file (output.txt) instead of part file

myfile.rdd
   .map(r =>  { val x = r.toString; x.substring(1, x.length-1)})
   .saveAsTextFile("C:/Users/rukku/Desktop/op")

example output (as part0001, part0002 in a folder)

    aaaaa,bbbb,ccccc,dddddd,eee
    e,f,g,h,i
    jj,kk,ll,mm,nn

Needed output (output.txt without a folder)

     aaaaabbbbcccccddddddeee
         e   f    g     h  i
        jj  kk   ll    mm nn

Answer 1

Following could be two step solution to get the expected outcome

1- A Metadata need to be defined which holds the length of each column in the target file. For every column it would require something like i) a string return function to add leading/trailing spaces to meet the length of a column ii) add leading zeros to a numeric column and so on.

2- Once the dataframe is loaded by applying all length fixing, .coalesce(1) can repartition the data to one and write single output file. But ensure that's not affecting the performance etc.

If the job is running very frequently and create too many files, a roll out job can be set up too too read and merge the files (This need to be designed based on the application behaviour, SLA etc. )