I am trying to run pyspark script. In this script i am executing sql query and creating dataframe after i am trying to change last column position into first but it is giving me error. Can you please help me how to do this?
code:
if(masterjobname.endswith('ADDR_Phani')):
df = sqlContext.sql(query)
print("Target: "+targetpath )
w = (Window().orderBy("SOURCE_COLUMN_VALUE"))
df = df.withColumn("SYSTEM_ID", dense_rank().over(w))
cols = df.columns.tolist()
cols = cols[-1:] + cols[:-1]
df = df[cols]
Okay i got the answer.
df.columns.values.tolist()
gives the list of the columns.
solution:
if(masterjobname.endswith('ADDR_Phani')):
df = sqlContext.sql(query)
print("Target: "+targetpath )
w = (Window().orderBy("SOURCE_COLUMN_VALUE"))
df = df.withColumn("SYSTEM_ID", dense_rank().over(w))
cols = df.columns.values.tolist()
cols = cols[-1:] + cols[:-1]
df = df[cols]
No matter if your variable is or isn't a list:
list(var)
You can use it as exception handling; ie ensure that it is indeed a list object.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.