具有对称列和索引（行）标签的Pandas数据框

Question

Problem Statement:

I have a variety of pandas dataframes that I would like to make symmetrical. Sometimes the row index labels will outnumber the col labels or vice-a-versa. In either case, both the row and column labels of the resulting dataframe should be the sorted union of all the labels. Any missing data would be filled in with np.nan .

My solution works, but it involves making 3 copies of the dataframe: the original df, the df with col labels filled out, and a df with row labels filled out. Any other solution I've tried results in an incompletely symmetrical matrix. I am looking for help to make my solution more simpler and more 'pythonic'.

Setup Asymmetrical Dataframe:

asym = pd.DataFrame.from_dict(  
         {'row': ['a','b','c','x','y','z','!'],
            'a': [ n, -.8,-.6,-.3, .8, .01,n],
            'b': [-.8,  n, .5, .7,-.9, .01,n],
            'c': [-.6, .5,  n, .3, .1, .01,n],
            'q': [-.3, .7, .3,  n, .2, .01,n],
            'r': [ .8,-.9, .1, .2,  n, .01,n],
            's': [ .01, .01, .01, .01,  .01, n,n],
       }).set_index('row')

Asymmetrical dataframe:

Notice the column labels are missing "x","y","z","!" and the row labels are missing "q","r","s" .

Attempt to make symmetrical:

df = asym
c = df.columns
r = df.index
label_union = set(c).union(set(r))

# fill rows with unique labels
df_1 = df.reindex(index=label_union.difference(r).union(set(r)), fill_value=n)
# fill cols with unique labels
df_2 = df_1.reindex(columns=label_union.difference(c).union(set(c)), fill_value=n)

# sort labels
df_2.sort_index(axis=0, inplace=True)
df_2.sort_index(axis=1, inplace=True)

The result below is right, but making three df copies seems unpythonic. I also want to perform the above code "inplace", as the dataframes I work with are large and numerous. Help me find a solution that gives the correct result below without all the df copies.

Symmetrical dataframe:

Note on use of "symmetrical": This resulting dataframe is not strictly symmetrical, meaning this matrix is not equal to its transpose. I am using "symmetrical" to refer specifically to the row and column labels. The matrix this toy example is emulating a genetic interaction matrix, where rows and columns are genes, and the corresponding value is a score depicting that interaction. To be truly symmetrical, the matrix would imply transitivity, which is not generally the case in genetic interactions.

Answer 1

You can reindex the both axes simultaneously with reindex :

label_union = asym.index.union(asym.columns)
asym = asym.reindex(index=label_union, columns=label_union)

The resulting output:

    !     a     b     c     q     r     s   x   y   z
! NaN   NaN   NaN   NaN   NaN   NaN   NaN NaN NaN NaN
a NaN   NaN -0.80 -0.60 -0.30  0.80  0.01 NaN NaN NaN
b NaN -0.80   NaN  0.50  0.70 -0.90  0.01 NaN NaN NaN
c NaN -0.60  0.50   NaN  0.30  0.10  0.01 NaN NaN NaN
q NaN   NaN   NaN   NaN   NaN   NaN   NaN NaN NaN NaN
r NaN   NaN   NaN   NaN   NaN   NaN   NaN NaN NaN NaN
s NaN   NaN   NaN   NaN   NaN   NaN   NaN NaN NaN NaN
x NaN -0.30  0.70  0.30   NaN  0.20  0.01 NaN NaN NaN
y NaN  0.80 -0.90  0.10  0.20   NaN  0.01 NaN NaN NaN
z NaN  0.01  0.01  0.01  0.01  0.01   NaN NaN NaN NaN

Answer 2

Here's a NumPy approach with np.ix_ that eases up creation of a 2D grid of valid indices and rest is just initializing with NaNs and assigning -

c = df.columns
r = df.index

L = np.union1d(c,r)
cols = np.searchsorted( L, c)
rows = np.searchsorted( L, r)
out = np.full((len(L),len(L)), np.nan)
out[np.ix_(rows, cols)] = df.values
df_out = pd.DataFrame(out, columns=L, index=L)

In terms of memory requirements, out would be a view into the output dataframe and as such won't occupy any additional memory.

Sample output -

In [556]: df_out
Out[556]: 
    !     a     b     c     q     r     s   x   y   z
! NaN   NaN   NaN   NaN   NaN   NaN   NaN NaN NaN NaN
a NaN   NaN -0.80 -0.60 -0.30  0.80  0.01 NaN NaN NaN
b NaN -0.80   NaN  0.50  0.70 -0.90  0.01 NaN NaN NaN
c NaN -0.60  0.50   NaN  0.30  0.10  0.01 NaN NaN NaN
q NaN   NaN   NaN   NaN   NaN   NaN   NaN NaN NaN NaN
r NaN   NaN   NaN   NaN   NaN   NaN   NaN NaN NaN NaN
s NaN   NaN   NaN   NaN   NaN   NaN   NaN NaN NaN NaN
x NaN -0.30  0.70  0.30   NaN  0.20  0.01 NaN NaN NaN
y NaN  0.80 -0.90  0.10  0.20   NaN  0.01 NaN NaN NaN
z NaN  0.01  0.01  0.01  0.01  0.01   NaN NaN NaN NaN

Answer 3

Get the union of the two indexes as you do today, then reindex twice the dataframe with 2 chained transpositions:

full_idx = asym.index.union(asym.columns)

asym.reindex(full_idx).T.reindex(full_idx).T
Out[116]: 
    !     a     b     c     q     r     s   x   y   z
! NaN   NaN   NaN   NaN   NaN   NaN   NaN NaN NaN NaN
a NaN   NaN -0.80 -0.60 -0.30  0.80  0.01 NaN NaN NaN
b NaN -0.80   NaN  0.50  0.70 -0.90  0.01 NaN NaN NaN
c NaN -0.60  0.50   NaN  0.30  0.10  0.01 NaN NaN NaN
q NaN   NaN   NaN   NaN   NaN   NaN   NaN NaN NaN NaN
r NaN   NaN   NaN   NaN   NaN   NaN   NaN NaN NaN NaN
s NaN   NaN   NaN   NaN   NaN   NaN   NaN NaN NaN NaN
x NaN -0.30  0.70  0.30   NaN  0.20  0.01 NaN NaN NaN
y NaN  0.80 -0.90  0.10  0.20   NaN  0.01 NaN NaN NaN
z NaN  0.01  0.01  0.01  0.01  0.01   NaN NaN NaN NaN