从数据框火花中删除一列

Question

I have a Spark dataframe with a very large number of columns. I want to remove two columns from it to get a new dataframe.

Had there been fewer columns, I could have used the select method in the API like this:

pcomments = pcomments.select(pcomments.col("post_id"),pcomments.col("comment_id"),pcomments.col("comment_message"),pcomments.col("user_name"),pcomments.col("comment_createdtime"));

But since picking columns from a long list is a tedious task, is there a workaround?

Answer 1

Use drop method and withColumnRenamed methods.

Example:

    val initialDf= ....

    val dfAfterDrop=initialDf.drop("column1").drop("coumn2")

    val dfAfterColRename= dfAfterDrop.withColumnRenamed("oldColumnName","new ColumnName")

Answer 2

Try this:

val initialDf = ...

val dfAfterDropCols = initialDf.drop("column1", "coumn2")