如何将不同列大小的 pandas dataframe 拆分为单独的数据帧？

Question

I have a large pandas dataframe, consisting of a different number of columns throughout the dataframe. Here is an example: Current dataframe example

I would like to split the dataframe into multiple dataframes, based on the number of columns it has.

Example output image here: Output image

Thanks.

Answer 1

If you have a dataframe of say 10 columns and you want to put the records with 3 NaN values in another result dataframe as those with 1 NaN , you can do this as follows:

# evaluate the number of NaNs per row
num_counts=df.isna().sum('columns')
# group by this number and add the grouped
# dataframe to a dictionary
results= dict()
num_counts=df.isna().sum('columns')
for key, sub_df in df.groupby(num_counts):
    results[key]= sub_df

After executing this code, results contains subsets of df where each subset contains the same number of NaN s (so the same number of non- NaN s).

If you want to write your results to a excel file, you just need to execute the following code:

with pd.ExcelWriter('sorted_output.xlsx') as writer:
    for key, sub_df in results.items():
        # if you want to avoid the detour of using dicitonaries
        # just replace the previous line by
        # for key, sub_df in df.groupby(num_counts):
        sub_df.to_excel(
            writer,
            sheet_name=f'missing {key}',
            na_rep='',
            inf_rep='inf',
            float_format=None,
            index=True,
            index_label=True,
            header=True)

Example:

# create an example dataframe
df=pd.DataFrame(dict(a=[1, 2, 3, 4, 5, 6], b=list('abbcac')))
df.loc[[2, 4, 5], 'c']= list('xyz')
df.loc[[2, 3, 4], 'd']= list('vxw')
df.loc[[1, 2], 'e']= list('qw')

It looks like this:

Out[58]: 
   a  b    c    d    e
0  1  a  NaN  NaN  NaN
1  2  b  NaN  NaN    q
2  3  b    x    v    w
3  4  c  NaN    x  NaN
4  5  a    y    w  NaN
5  6  c    z  NaN  NaN

If you execute the code above on this dataframe, you get a dictionary with the following content:

0:    a  b  c  d  e
   2  3  b  x  v  w

1:    a  b  c  d    e
   4  5  a  y  w  NaN

2:    a  b    c    d    e
   1  2  b  NaN  NaN    q
   3  4  c  NaN    x  NaN
   5  6  c    z  NaN  NaN

3:    a  b    c    d    e
   0  1  a  NaN  NaN  NaN

The keys of the dictionary are the number of NaN s in the row and the values are the dataframes which contain only rows with that number of NaN s in them.

Answer 2

If I'm getting you right, what you want to do is to split existing 1 dataframe with n columns into ceil(n/5) dataframes, each with 5 columns, and the last one with the reminder of n/5 .

If that's the case this will do the trick:

import pandas as pd
import math

max_cols=5

dt={"a": [1,2,3], "b": [6,5,3], "c": [8,4,2], "d": [8,4,0], "e": [1,9,5], "f": [9,7,9]}

df=pd.DataFrame(data=dt)

dfs=[df[df.columns[max_cols*i:max_cols*i+max_cols]] for i in range(math.ceil(len(df.columns)/max_cols))]

for el in dfs:
    print(el)

And output:

   a  b  c  d  e
0  1  6  8  8  1                                            
1  2  5  4  4  9                                            
2  3  3  2  0  5                                               
   f                                                        
0  9                                                        
1  7                                                        
2  9                                                        

[Program finished]