Unable to add worksheets to a xlsx file in Python

Question

I am trying to export data by running dynamically generated SQLs and storing the data into dataframes which I eventually export into an excel sheet. However, through I am able to generate the different results by successfully running the dynamic sqls, I am not able to export it into different worksheets within the same excel file. It eventually overwrites the previous result with the last resultant data.

for func_name in df_data['FUNCTION_NAME']:
        sheet_name = func_name  
        sql = f"""select * from table({ev_dwh_name}.OVERDRAFT.""" + sheet_name + """())"""
        print(sql)                
        dft_tf_data = pd.read_sql(sql,sf_conn)
        print('dft_tf_data')  
        print(dft_tf_data)
        # dft.to_excel(writer, sheet_name=sheet_name, index=False)
        
        with tempfile.NamedTemporaryFile('w+b', suffix='.xlsx', delete=False) as fp:
            #dft_tf_data.to_excel(writer, sheet_name=sheet_name, index=False)
            print('Inside Temp File creation')
            temp_file = path + f'/fp.xlsx'
            writer = pd.ExcelWriter(temp_file, engine = 'xlsxwriter')
            dft_tf_data.to_excel(writer, sheet_name=sheet_name, index=False)
            writer.save()
    print(temp_file) 

I am trying to achieve the below scenario.

1. Based on the FUNCTION_NAME, it should add a new sheet in the existing excel and then write the data from the query into the worksheet.
2. The final file should have all the worksheets.

Is there a way to do it. Please suggest.

Answer 1

I'd only expect a file not found that to happen once (first run) if fp.xlsx doesn't exist. fp.xlsx gets created on the line

writer= 

if it doesn't exist and since the line is referencing that file it must exist or the file not found error will occur. Once it exists then there should be no problems.

I'm not sure of the reasoning of creating a temp xlsx file. I dont see why it would be needed and you dont appear to use it.

The following works fine for me, where fp.xlsx initially saved as a blank workbook before running the code.

sheet_name = 'Sheet1'
with tempfile.NamedTemporaryFile('w+b', suffix='.xlsx', delete=False) as fp:
    print('Inside Temp File creation')
    temp_file = path + f'/fp.xlsx'
    writer = pd.ExcelWriter(temp_file,
                            mode='a',
                            if_sheet_exists='overlay',
                            engine='openpyxl')
    dft_tf_data.to_excel(writer,
                 sheet_name=sheet_name,
                 startrow=writer.sheets[sheet_name].max_row+2,
                 index=False)
    writer.save()
print(temp_file)