handle large pivot tables in Pandas

Question

I'm creating a pretty large pivot table, but in the Python IDE or notebook I can see only partial results.

I would like to have all the results in a better format, such as an excel sheet.

but if a try to export results to excel I can only see the D computed field, not the data from A and C variable.

do you have an idea on better managing large pivot table in pandas?

import numpy as np
import pandas as pd

N = 100
df = pd.DataFrame({
    'A': pd.date_range(start='2016-01-01',periods=N,freq='D'),
    'x': np.linspace(0,stop=N-1,num=N),
    'y': np.random.rand(N),
    'C': np.random.choice(['Low','Medium','High'],N).tolist(),
    'D': np.random.normal(100, 10, size=(N)).tolist()
    })

my_table=pd.pivot_table(df,index=["A","C"],values=["D"], aggfunc='sum',margins=True)


writer = pd.ExcelWriter('C:/pandas_positioning.xlsx', engine='xlsxwriter')


my_table.to_excel(writer, sheet_name='my_data',
         startrow=7, startcol=4, header=False, index=False)

writer.save()

Answer 1

This should fix your problem

my_table.to_excel(writer, sheet_name='my_data',
     startrow=7, startcol=4, header=False)

A and C are index and you are not writing them to file.