How to pickle multiple pandas Dataframes and concatenate all of them in single command

Question

Files =['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H']
fout='/PATH/df/{}/{}.F.K.df'.format('train',Files[0])
df1=pd.read_pickle(fout)
df1 = df1[df1.columns[:100]]

fout='/PATH/df/{}/{}.F.K.df'.format('train',Files[1])
df2=pd.read_pickle(fout)
df2 = df2[df2.columns[:100]]

fout='/PATH/df/{}/{}.F.K.df'.format('train',Files[2])
df3=pd.read_pickle(fout)
df3 = df3[df3.columns[:100]]

fout='/PATH/df/{}/{}.F.K.df'.format('train',Files[3])
df4=pd.read_pickle(fout)
df4 = df4[df4.columns[:100]]

fout='/PATH/df/{}/{}.F.K.df'.format('train',Files[4])
df5=pd.read_pickle(fout)
df5 = df5[df5.columns[:100]]

fout='/PATH/df/{}/{}.F.K.df'.format('train',Files[5])
df6=pd.read_pickle(fout)
df6 = df6[df6.columns[:100]]

fout='/PATH/df/{}/{}.F.K.df'.format('train',Files[6])
df7=pd.read_pickle(fout)
df7 = df7[df7.columns[:100]]

fout='/PATH/df/{}/{}.F.K.df'.format('train',Files[7])
df8=pd.read_pickle(fout)
df8 = df8[df8.columns[:100]]

df = pd.concat([df1, df2, df3, df4, df5, df6, df7, df8], axis = 1)
df = df.loc[:,~df.columns.duplicated()]

I have these following commands and the first 8 blocks are all repeated codes with very small modifications. Is there a way I can do this for doing something like this:

[pd.read_pickle('/PATH/df/{}/{}.FKdf'.format('train',Files[i])) for i in Files]

But then this just gives me many dataframes that'll be set to the same variable and I don't know how to map it to 8 different dataframes and then concat them all in one go.

Answer 1

Your problem is that you are creating an arbitrary number of variables, which you don't need. Use Files to read the DataFrames into a list, which you then concatenate.

df = pd.concat([pd.read_pickle('/PATH/df/{}/{}.F.K.df'.format('train', f)).iloc[:, :100] 
                for f in Files], 
               axis=1)