Display missing values of specific column based on another specific column

Question

this is my problem

Let's say I have 2 columns on the dataframe which look like this:

 Type   | Killed
_______ |________
 Dog        1
 Dog       nan
 Dog       nan
 Cat        4
 Cat       nan
 Cow        1
 Cow       nan

I would like to display all missing value in Killed according to the Type and count them

My desire outcome would look something like this:

Type | Sum(isnull)
Dog       2
Cat       1
Cow       1

Is there anyway to display this?

Answer 1

You can use boolean indexing with value_counts :

print (df.ix[df.Killed.isnull(), 'Type'].value_counts().reset_index(name='Sum(isnull)'))

  index  Sum(isnull)
0   Dog            2
1   Cow            1
2   Cat            1

Or aggregate size , it seems faster:

print (df[df.Killed.isnull()]
            .groupby('Type')['Killed']
            .size()
            .reset_index(name='Sum(isnull)'))

  Type  Sum(isnull)
0  Cat           1
1  Cow           1
2  Dog           2

Timings :

df = pd.concat([df]*1000).reset_index(drop=True)

In [30]: %timeit (df.ix[df.Killed.isnull(), 'Type'].value_counts().reset_index(name='Sum(isnull)'))
100 loops, best of 3: 5.36 ms per loop

In [31]: %timeit (df[df.Killed.isnull()].groupby('Type')['Killed'].size().reset_index(name='Sum(isnull)'))
100 loops, best of 3: 2.02 ms per loop

Answer 2

I can get you both isnull and notnull

isnull = np.where(df.Killed.isnull(), 'isnull', 'notnull')
df.groupby([df.Type, isnull]).size().unstack()