如何在scala中的空数据框现有列上添加赋值？

Question

I am reading a csv file which has | delimiter at last , while load method make last column in dataframe with no name and no values in Spark 1.6

df.withColumnRenamed(df.columns(83),"Invalid_Status").drop(df.col("Invalid_Status"))

val df = sqlContext.read.format("com.databricks.spark.csv").option("delimiter","|").option("header","true").load("filepath") 
val df2 = df.withColumnRenamed(df.columns(83),"Invalid_Status").

I expected result 
root
 |-- FddCell: string (nullable = true)
 |-- Trn_time: string (nullable = true)
 |-- CELLNAME.FddCell: string (nullable = true)
 |-- Invalid_Status: string (nullable = true)

but actual output is
root
 |-- FddCell: string (nullable = true)
 |-- Trn_time: string (nullable = true)
 |-- CELLNAME.FddCell: string (nullable = true)
 |-- : string (nullable = true)

with no value in column so I have to drop this column and again make new column.

Answer 1

It is not completely clear what you want to do, to just rename the column to Invalid_Status or to drop the column entirely. What I understand is, you are trying to operate (rename/drop) on the last column which has no name.

But I will try to help you with both the solution -

To Rename the column with same values (blanks) as it is:

val df2 = df.withColumnRenamed(df.columns.last,"Invalid_Status")

Only To Drop the last column without knowing its name, use:

val df3 = df.drop(df.columns.last)

And then add the "Invalid_Status" column with default values:

val requiredDf = df3.withColumn("Invalid_Status", lit("Any_Default_Value"))