如何在没有Jupyter Notebook崩溃的情况下显示DataFrame？

Question

Everytime I try to display a complete DataFrame in my jupyter notebook, the note book crashes. The file wont start up so I had to make a new jupyter notebook file. When i do display(df) it only shows a couple of the rows, when I need to show 57623 rows. I need to show results for all of these rows and put them into an html file.

I tried setting the max rows and max columns, but the entire dataframe would not print out without the notebook crashing

''' python pd.set_option('display.max_columns', 24) "pd.set_option('display.max_rows', 57623) '''

The expected results were for the entire DataFrame to print out, but instead the notebook would have an hourglass next to it and nothing would load.

Answer 1

I think your machine is not powerful enough to handle so much data. I can display 57623 rows for my data (with 16 columns) without any problem. But my machine has 252GB memory, and it still took about 1 minute to display these data in jupyter notebook.

If I tried to scroll through the data, it's slow, and sometimes stuck for a while.

On the other hand, what do you want to achieve by displaying all data here? There's definitely another way to achieve the same thing as you are doing now.