Function 获取火花列表 dataframe 并转换为 pandas 然后 Z628CB5675FF524F3EZ19BAA7FAA

Question

import pyspark
            
dfs=[df1,df2,df3,df4,df5,df6,df7,df8,df9,df10,df1,df12,df13,df14,df15]
    
for x in dfs:
    y=x.toPandas()
    y.to_csv("D:/data")

This is what I wrote, but I actually want the function to take this list and convert every df into a pandas df and then convert it to csv and save it in the order as it appears on dfs list and save it to a particular directory in the order of name. Is there a possible way to write such function? PS D:/data is just an imaginary path and is used for explanation.

Answer 1

If you will convert a dataframe to a csv, you still need to state it in df.to_csv . So, try:

for x in dfs:
       y=x.toPandas()
       y.to_csv(f"D:/data/df{dfs.index(x) + 1}.csv")

I set it as df{dfs.index(x) + 1} so that the file names will be df1, df2, ... etc.