Calling variables with f-string inside concat for loop

Question

I am trying to combine dataframes using pd.concat .

I have 7 models, divided into 2 depending on year (2021 and 2022) so in total I have 14 dataframes each containing 4 columns. They can be created by the following command:

concat_list = ['expert_2021', 'expert_2022', 'forecast_168_2021', 'forecast_168_2022', 
               'forecast_24_2021', 'forecast_24_2022', 'forecast_custom_2021', 'forecast_custom_2022', 
               'forecast_lear_2021', 'forecast_lear_2022', 'forecast_standard_2021',
               'forecast_standard_2022', 'auto_2021', 'auto_2022'] 
n = 14
df_list = [pd.DataFrame({"Price_REG1":[], "Price_REG2":[], "Price_REG3":[], "Price_REG4":[]}) for x in range(n)]

for i, j in zip(concat_list, range(14)):
    locals()[i] = df_list[j]

Now, I want to combine these into 8 new dataframes, each representing 1 year and 1 column so 2 years * 4 columns = 8. I want to do this in a for loop. I am using f-strings to loop over the years and the columns to place the dataframes inside a list.

year_list = [2021, 2022]
prediction = []
for p in year_list:
    for j, s in zip(range(1,5), range(4)):
        a = pd.concat([f'forecast_24_{p}.Price_REG{j}', f'forecast_168_{p}.Price_REG{j}', 
                       f'forecast_standard_{p}.Price_REG{j}', f'forecast_custom_{p}.Price_REG{j}', 
                       f'expert_{p}.Price_REG{j}', f'forecast_lear_{p}.Price_REG{j}', 
                       f'auto_{p}.Price_REG{j}'], axis=1)
        prediction.append(a)

This gives me TypeError: cannot concatenate object of type '<class 'str'>'; only Series and DataFrame objs are valid TypeError: cannot concatenate object of type '<class 'str'>'; only Series and DataFrame objs are valid

So, I understand that the strings is the problem. But my question is if there is a way to make these strings to call the dataframes, or if there is some other alternative solution to this kind of problem?

Thank you.

Answer 1

You have to use globals() to get your dataframes:

year_list = [2021, 2022]
prediction = []
for p in year_list:
    for j, s in zip(range(1,5), range(4)):
        df_list2 = [globals()[f'{prefix}_{p}'][f'Price_REG{j}']
                        for prefix in ['forecast_24', 'forecast_168', 'forecast_standard', 
                                       'forecast_custom', 'expert', 'forecast_lear', 'auto']]
                
        a = pd.concat(df_list2, axis=1)
        prediction.append(a)

Note : you can't use locals() here because the scope is the list comprehension and you won't be able to access your dataframes.