Python data frame apply filter on multiple columns with same condition?

Question

Here is my pandas data frame.

new_data =

    name   duration01        duration02          orz01      orz02     
    ABC   1 years 6 months    5 months           Nan        Google     
    XYZ   4 months            3 years 2 months   Google      Zensar      
    TYZ   4 months            4 years            Google In   Google   
    OPI   2 months            3 months           Nan       accenture    
    NRM   9 months            3 years            Google      Zensar     

I want to find out the name of employees who works in Google and there duration in months.Here the value contains in the multiple columns ? How to apply filter on multiple columns ?

duration01 => orz01 ( how many months/years employee spend in orz01) duration02 => orz02 ( how many months/years employee spend in orz02)

There are total 10 orz and 10 respective duration columns.

I tried below code

# Selected the required columns
orz_cols = new_data.columns[new_data.columns.str.contains('orz')]

new_data [ new_data[orz_cols].apply(lambda x: x.str.contains('Google')) ]

But its not printing proper data ?

How do I achieve this

I want output like below

name  Total_duration_in Google_in_Months
ABC   5 months
XYZ   4 months
TYZ   52 months  

Using the first part what @Stefan gave I did below part to convert years to months

# filter the data

Google_Data = dt1[dt1['orz'].str.contains('Google')]

dur = []

for i in range(0,len(Google_Data['duration'])):
    dur.append(Google_Data['duration'][i].split())

months_list = []

for i in range(0,len(dur)):
    #print dur[i]
    if dur[i][1] == 'years':
        if len(dur[i]) > 2:
            val1 = int(dur[i][0]) * 12 + int(dur[i][2])
            val11 = str(val1)+" months"
            months_list.append(val11)
        else:
            val2 = int(dur[i][0]) * 12
            val22 = str(val2)+" months"
            months_list.append(val22)
    else:
        val3 = dur[i][0]+" months"
        months_list.append(val3)

months_list[:3]

# Concat
df2 = pd.DataFrame(months_list,index=Google_Data.index.copy())

Google_duration = pd.concat([Google_Data, df2], axis=1)


Output :

                    organization                      Duration_In_Months
name        
Aparna Arora        Google Headstrong Capital Markets   60 months
Aparna Dasgupta     Google                              24 months
Aparna Dhar         Google India Ltd                    56 months

Now I want to perform final step ie take the sum by grouping the name but here 'name' is index. I am struggling to get the sum.

Here what i am trying

# Splitting the Duration_In_Months to get only number values
# Its returning the type as 'str'

Google_duration1 = Google_duration.Duration_In_Months.apply(lambda x : x.split()[0])

# apply groupby

Genpact_dur2.index.groupby(Genpact_dur2['Duration_In_Months'])

How do I Groupby index and take the sum ?

Thanks,

Answer 1

You could do as follows:

Set index and get columns to combine:

df.set_index('name', inplace=True)    
orz_cols = [col for col in df.columns if col.startswith('orz')]
duration_cols = [col for col in df.columns if col.startswith('duration')]
merge_cols = zip(orz_cols, duration_cols)

Use pd.concat() to reshape and rename:

long_df = pd.concat([df.loc[:, cols].rename(columns={col: col[:-2] for col in orz_cols + duration_cols}) for cols in merge_cols])

Eliminate non-Google orz entries:

long_df = long_df[long_df.orz.str.contains('Google')]

Calculcate duration depending on month & year :

long_df.duration = long_df.duration.str.split().apply(lambda x: int(x[0]) if x[1] == 'months' else int(x[0]) * 12)

Sum by name :

long_df.groupby(level='name')['duration'].sum()
      duration
name          
ABC          5
NRM          9
TYZ         52
XYZ          4

Answer 2

Consider reshaping using pandas.melt , then conditionally parsing out values for years and months using np.where() . Finally, aggregate by the Google organization.

import pandas as pd
import numpy as np

...
# LIST OF SUBSET COLUMNS
durationCols = [c for c in df.columns if 'duration' in c ]
orzCols = [c for c in df.columns if 'orz' in c ]

# MELT AND MERGE
df = pd.merge(pd.melt(df, id_vars=['name'], value_vars=durationCols,
                  var_name=None, value_name='duration'),
              pd.melt(df, id_vars=['name'], value_vars=orzCols,
                  var_name=None, value_name='orz'),
              right_index=True, left_index=True, on=['name'])[['name', 'duration', 'orz']]

# DURATION CONDITIONAL CALCULATION (YEAR + MONTH)
df['actual_dur'] = np.where(df['duration'].str.contains('year'),
                            df['duration'].str[:1], 0).astype(int) * 12 + \
                   np.where(df['duration'].str.contains('year.*month'),
                            df['duration'].str[8:9],
                            np.where(df['duration'].str.contains('month'),
                                     df['duration'].str[:1], 0)).astype(int)

df['orz'] = np.where(df['orz']\
                     .str.contains('Google'), 'Google', df['orz'])    

# SUM DURATION AND OUTPUT DF
df = df[df['orz']=='Google'].groupby(['name','orz']).sum().reset_index()    
df = df[['name','actual_dur']]
df.columns = ['name', 'Total_duration_in Google_in_Months']

Output

#   name  Total_duration_in Google_in_Months
# 0  ABC                                   5
# 1  NRM                                   9
# 2  TYZ                                  52
# 3  XYZ                                   4