PySpark Dataframe 的最佳实践 - 删除多列？

Question

Let's say one wants to drop a column from a dataframe. Can that be done without creating a new dataframe?

df = df.drop("tags_s")

It seems like creating a new dataframe is safer, more correct, is that correct? What problems might one run into by re-using a dataframe as above?

If re-using a dataframe is a bad practice, let's say one wants to drop several columns that match a pattern:

for col in df.columns:
  if col.startswith("aux_"):
    df = df.drop(col)

Creating a new dataframe each time seems impractical in this case. What is the best practice?

Answer 1

If you are going to drop multiple columns, I'd say that the first step is identify the columns, save it in a list and then do a single .drop , something like:

your_column_list = [col for col in df.columns if col.startswith("aux") ]
df.drop(*your_column_list)

Answer 2

This is according to my understanding with spark dataframe, you don't have to worry about it returning new dataframe each time, what you're doing there is just a transformation on the dataframe.You can perform many transformations to the dataframe, it's only when you perform any action against the dataframe that's when it's produce a new dataframe.

Here is more information on transformations vs actions, https://spark.apache.org/docs/latest/rdd-programming-guide.html#basics