pandas groupby and percentage of occurrences of each value of a column

Question

I have a pandas dataframe like this and want to create a column like created_column :

       iv_1  iv_2  iv_3  iv_4  iv_5  col2rplc  created_column
0       0      0     0     0     0      a          0
333     0      0     0     0     0      b          0
      ......
222     1      2     3     4     5      aa         1
324     1      2     3     4     5      cc         1
      ......
1234    1      0     0     0     1      a          1
1235    0      2     0     4     0      a          0
1236    0      0     3     0     0      a          0
1237    0      0     1     0     0      b          0
1238    0      2     0     2     0      b          0
1239    3      0     0     0     3      b          1

explanation:
I want to create a column that will have 1 in rows where values in iv_5 column has occurred for less than or equal to 40% of the data, that would be for rows with values 1, 3 & 5, as shown in above example. how do i do this?

second question:
How do I also include less than x% and greater than y%, in creation of other column, as similar to above column creation.

Answer 1

Use GroupBy.transform with divide length of DtaFrame and test by Series.le for less or equal:

df['created_column'] = df.groupby('iv_5')['iv_5'].transform('size').div(len(df)).le(0.4).view('i1')
print (df)
      iv_1  iv_2  iv_3  iv_4  iv_5 col2rplc  created_column
0        0     0     0     0     0        a               0
333      0     0     0     0     0        b               0
222      1     2     3     4     5       aa               1
324      1     2     3     4     5       cc               1
1234     1     0     0     0     1        a               1
1235     0     2     0     4     0        a               0
1236     0     0     3     0     0        a               0
1237     0     0     1     0     0        b               0
1238     0     2     0     2     0        b               0
1239     3     0     0     0     3        b               1

Or:

s = df['iv_5'].value_counts(normalize=True)
idx = s.index[s <= 0.4]

df['created_column'] = df['iv_5'].isin(idx).view('i1')

If need Series.between , both are inclusive by default, it means >= , <= , for > and < use parameter inclusive=False :

df['created_column'] = df.groupby('iv_5')['iv_5'].transform('size').div(len(df)).between(0.2, 0.5).view('i1')
print (df)

      iv_1  iv_2  iv_3  iv_4  iv_5 col2rplc  created_column
0        0     0     0     0     0        a               0
333      0     0     0     0     0        b               0
222      1     2     3     4     5       aa               1
324      1     2     3     4     5       cc               1
1234     1     0     0     0     1        a               0
1235     0     2     0     4     0        a               0
1236     0     0     3     0     0        a               0
1237     0     0     1     0     0        b               0
1238     0     2     0     2     0        b               0
1239     3     0     0     0     3        b               0

If need combination like > and <= between cannot be used, here is alternative:

s1 = df.groupby('iv_5')['iv_5'].transform('size').div(len(df))
df['created_column'] = ((s1 > 0.2) & (s1 <= 0.6)).view('i1')

print (df)
      iv_1  iv_2  iv_3  iv_4  iv_5 col2rplc  created_column
0        0     0     0     0     0        a               1
333      0     0     0     0     0        b               1
222      1     2     3     4     5       aa               0
324      1     2     3     4     5       cc               0
1234     1     0     0     0     1        a               0
1235     0     2     0     4     0        a               1
1236     0     0     3     0     0        a               1
1237     0     0     1     0     0        b               1
1238     0     2     0     2     0        b               1
1239     3     0     0     0     3        b               0