如何在pyspark中重命名数据框的列名？

Question

I want to rename one column name from dataframe columns, So currently the Column name is rate%year . I want to rename it as rateyear in pyspark .

Possibly, we can rename columns at dataframe and table level after registering dataframe as table, but at table level "%" will create problem so i want to rename at dataframe level itelf.

I tried this- data.selectExpr("rate%year as rateyear")

but getting this error pyspark.sql.utils.AnalysisException: u"cannot resolve 'rate' given input columns

Thanks.

Answer 1

Try this:

sqlContext.registerDataFrameAsTable(data, "myTable")
data = sqlContext.sql("SELECT rate%year AS rateyear from myTable")

Answer 2

I wrote an easy and fast function for you to remove % from column names. Enjoy! :)

def rename_cols(rename_df):
    for column in rename_df.columns:
        new_column = column.replace('%','')
        rename_df = rename_df.withColumnRenamed(column, new_column)
    return rename_df

Answer 3

Possible way of renaming at dataframe level-

oldColumns=['rate%year']
newColumns = ["rateyear"]
df1 = reduce(lambda df, idx: df.withColumnRenamed(oldColumns[idx], newColumns[idx]), xrange(len(oldColumns)), df)

this is working fine at dataframe level. any suggestion how to resolve at table level?

Answer 4

Simple and quick way to alter dataframe column names.

def format_col(df):    
    cols = [col.replace("%", "") for col in df.columns]
    res_df = df.toDF(*cols)
    return res_df