pandas display: truncate column display rather than wrapping

Question

With lengthy column names, DataFrames will display in a very messy form seemingly no matter what options are set.

Info: I'm in Jupyter QtConsole, pandas 0.20.1, with the following relevant options specified at startup:

pd.set_option('display.max_colwidth', 20)
pd.set_option('expand_frame_repr', False)
pd.set_option('display.max_rows', 25)

Question: how can I truncate the DataFrame if necessary rather than wrapping the columns to the next line, while keeping expand_frame_repr=False ?

Here's an example. Again, the issue doesn't depend on the number of columns but length of the columns.

This will not cause an issue:

df = pd.DataFrame(np.random.randn(1000, 1000),
                  columns=['col' + str(i) for i in range(1000)])

As the output is perfectly readable and looks like:

The same DataFrame with long column names causes the issue I'm talking about:

df = pd.DataFrame(np.random.randn(1000, 1000),
                  columns=['very_long_col_name_' 
                           + str(i) for i in range(1000)])

Is there any way to conform the second output to be like the first that I'm missing? (Through specifying an option, not through using .iloc every time I want to view.)

Answer 1

Use max_columns

from string import ascii_letters

df = pd.DataFrame(np.random.randint(10, size=(5, 52)), columns=list(ascii_letters))

with pd.option_context(
    'display.max_colwidth', 20,
    'expand_frame_repr', False,
    'display.max_rows', 25,
    'display.max_columns', 5,
):
    print(df.add_prefix('really_long_column_name_'))

   really_long_column_name_a  really_long_column_name_b            ...              really_long_column_name_Y  really_long_column_name_Z
0                    8                          1                  ...                                1                          9      
1                    8                          5                  ...                                2                          1      
2                    5                          0                  ...                                9                          9      
3                    6                          8                  ...                                0                          9      
4                    1                          2                  ...                                7                          1      

[5 rows x 52 columns]

---

Another idea... Obviously not exactly what you want, but maybe you can twist it to your needs.

d1 = df.add_suffix('_really_long_column_name')

with pd.option_context('display.max_colwidth', 4, 'expand_frame_repr', False):
    mw = pd.get_option('display.max_colwidth')
    print(d1.rename(columns=lambda x: x[:mw-3] + '...' if len(x) > mw else x))

   a...  b...  c...  d...  e...  f...  g...  h...  i...  j...  ...   Q...  R...  S...  T...  U...  V...  W...  X...  Y...  Z...
0    6     5     5     5     8     3     5     0     7     6   ...     9     0     6     9     6     8     4     0     6     7 
1    0     5     4     7     2     5     4     3     8     7   ...     8     1     5     3     5     9     4     5     5     3 
2    7     2     1     6     5     1     0     1     3     1   ...     6     7     0     9     9     5     2     8     2     2 
3    1     8     7     1     4     5     5     8     8     3   ...     3     6     5     7     1     0     8     1     4     0 
4    7     5     6     2     4     9     7     9     0     5   ...     6     8     1     6     3     5     4     2     3     2 

Answer 2

Looks like it will need an enhancement. The relevant code in the repr function appears to be here:

    max_rows = get_option("display.max_rows")
    max_cols = get_option("display.max_columns")
    show_dimensions = get_option("display.show_dimensions")
    if get_option("display.expand_frame_repr"):
        width, _ = console.get_console_size()
    else:
        width = None
    self.to_string(buf=buf, max_rows=max_rows, max_cols=max_cols,
                   line_width=width, show_dimensions=show_dimensions)

So either you pass expand_frame_repr=True and it wraps on the line width, or you pass expand_frame_repr=False and it shouldn't. But it looks like there is a bug in the code (this should be pandas 0.20.3 iirc):

in pd.io.formats.format.DataFrameFormatter :

def _chk_truncate(self):
    """
    Checks whether the frame should be truncated. If so, slices
    the frame up.
    """
    from pandas.core.reshape.concat import concat

    # Column of which first element is used to determine width of a dot col
    self.tr_size_col = -1

    # Cut the data to the information actually printed
    max_cols = self.max_cols
    max_rows = self.max_rows

    if max_cols == 0 or max_rows == 0:  # assume we are in the terminal
                                        # (why else = 0)
        (w, h) = get_terminal_size()
        self.w = w
        self.h = h
        if self.max_rows == 0:
            dot_row = 1
            prompt_row = 1
            if self.show_dimensions:
                show_dimension_rows = 3
            n_add_rows = (self.header + dot_row + show_dimension_rows +
                          prompt_row)
            # rows available to fill with actual data
            max_rows_adj = self.h - n_add_rows
            self.max_rows_adj = max_rows_adj

        # Format only rows and columns that could potentially fit the
        # screen
        if max_cols == 0 and len(self.frame.columns) > w:
            max_cols = w
        if max_rows == 0 and len(self.frame) > h:
            max_rows = h

Looks like it intended to do what you wanted, but was unfinished. It's checking max_cols against the number of columns, not the total width of the columns.

So you could either create a show_df function that would calculate the correct number of columns and show it in an option_context like pi2Squared's answer, or fix it here (and maybe submit a patch if you need it distributed).

Answer 3

As others have pointed out, Pandas itself seems to be bugged or badly designed here, so a workaround is required.

Most of the time this problem occurs with numerical columns, since numbers are relatively short. Pandas will split the column heading onto multiple lines if there are spaces in it, so you can "hack in" the correct behavior by inserting spaces into column headings for numerical columns when you display the dataframe. I have a one-liner to do this:

def colfix(df, L=5): return df.rename(columns=lambda x: ' '.join(x.replace('_', ' ')[i:i+L] for i in range(0,len(x),L)) if df[x].dtype in ['float64','int64'] else x )

do display your dataframe, simply type

colfix(your_df)

note that the renaming is not going to permanently change the dataframe, it will only add spaces to the names for the purposes of displaying it that one time.

Results (in a Jupyter Notebook):

With colfix :

Without: