python dataframe as function input and get another dataframe with new name as output

Question

I have a dataframe df with lots of processing on different rows and columns. Eventually I'd like to get a new df called eg processed_df . This is what I have done:

import pandas as pd
import numpy as np

def foofunc(df):
    name =[x for x in globals() if globals()[x] is df][0] # get df name as string
    output_df='processed_'+str(name)
    
    output_df=df.head(2) # e.g as process, in reality is ~ 50 operations
    print(f'output dataframe name is: {str(output_df)})') #expect to get: processed_df
    return output_df

testdf = pd.DataFrame(np.random.randint(0,100,size=(5, 2)), columns=list('AB'))
foofunc(testdf) # expect to get processed_testdf

processed_df

Then here in the last line, I get the error:

NameError: name 'processed_df' is not defined

To be more clear, this is part of a pipeline, so I'd like just to give a df and get out the processed with a desired name. In general, is my approach a good practice to do such operations on dataframes?

Thank you!

Answer 1

I don't see a good reason to have a function auto-generate a name and put its result into the global namespace, when python already binds function results to names. After that name has been generated, how would another piece of code know what it is called? And suppose that input df wasn't in the function's global namespace and its global name (or one of its global names if it has multiple references) can't be found?

There are many ways to write a pipeline, the easiest being

df = do_thing_1(df)
df = do_thing_2(df)
...

This has the advantage that the caller gets to decide the name. And it gets rid of intermediate dataframes that are otherwise consuming memory.

That said, your problem is that you don't assign the result back to the global namespace... and you use the wrong name for the generated dataframe (getting back to that "how do you know what the name is" problem). A solution is

import pandas as pd
import numpy as np

def foofunc(df):
    name =[x for x in globals() if globals()[x] is df][0] # get df name as string
    output_df_name='processed_'+str(name)
    
    output_df=df.head(2) # e.g as process, in reality is ~ 50 operations
    print(f'output dataframe name is: {str(output_df)})') #expect to get: processed_df
    globals()[output_df_name] = output_df

testdf = pd.DataFrame(np.random.randint(0,100,size=(5, 2)), columns=list('AB'))
foofunc(testdf) # expect to get processed_testdf

processed_testdf