`re.sub()` in pandas

Question

Say I have:

s = 'white male, 2 white females'

And want to "expand" this to:

'white male, white female, white female'

A more complete list of cases would be:

• 'two hispanic males, two hispanic females'
  • --> 'hispanic male, hispanic male, hispanic female, hispanic female'
• '2 black males, white male'
  • --> 'black male, black male, white male'

It seems like I am close with:

import re

# Do I need boundaries here?
mult = re.compile('two|2 (?P<race>[a-z]+) (?P<gender>(?:fe)?male)s')

# This works:
s = 'white male, 2 white females'
mult.sub(r'\g<race> \g<gender>, \g<race> \g<gender>', s)
# 'white male, white female, white female'

# This fails:
s = 'two hispanic males, 2 hispanic females'
mult.sub(r'\g<race> \g<gender>, \g<race> \g<gender>', s)
# ' ,  , hispanic males, hispanic female, hispanic female,'

What is creating the trip-up in the second case?

Bonus question: Is there a method of pandas' Series that implements this functionality directly instead of using Series.apply() ? Sorry to revise my question and waste anyone's time here.

For instance, on:

s = pd.Series(
    ['white male',
     'white male, white female',
     'hispanic male, 2 hispanic females',
     'black male, 2 white females'])

Is there a faster route than:

s.apply(lambda x: mult.sub(..., x))

Answer 1

With regards to your "bonus" question, you can use pandas.Series.str.replace , which is part of the pandas.Series.str methods which work with regex:

In [10]: import re

In [11]: import pandas as pd

In [12]: s = pd.Series(
    ...:     ['white male',
    ...:      'white male, white female',
    ...:      'hispanic male, 2 hispanic females',
    ...:      'black male, 2 white females'])

In [13]: mult = re.compile('two|2 (?P<race>[a-z]+) (?P<gender>(?:fe)?male)s')
    ...:

In [14]: s.str.replace(mult, r'\g<race> \g<gender>, \g<race> \g<gender>')
Out[14]:
0                                         white male
1                           white male, white female
2    hispanic male, hispanic female, hispanic female
3             black male, white female, white female
dtype: object

Whether or not these methods are significantly faster than .apply I don't know. I suspect that you'll never be very fast working with object dtypes.

Note, if found this issue regarding these methods being on the slow side. I suppose until they decide it is worth it to write out a Cythonized implementation then you probably can't hope for much.

Answer 2

IIUC, you need to put paranthesis around two|2 like (two|2) if you want to match either.

import re

mult = re.compile('(two|2) (?P<race>[a-z]+) (?P<gender>(?:fe)?male)s')
s = 'two hispanic males, 2 hispanic females'
mult.sub(r'\g<race> \g<gender>, \g<race> \g<gender>', s)
# 'hispanic male, hispanic male, hispanic female, hispanic female'

Answer 3

Regarding your regex itself I'd go with the following one which is more general and optimized.

In [14]: mult = re.compile('(?:two|2) ([^,]+)')

In [15]: s = 'two hispanic males, 2 hispanic females'

In [16]: mult.sub(lambda x: x.group(1) + ' ' + x.group(1), s)
Out[16]: 'hispanic males hispanic males, hispanic females hispanic females'

But about the performance and applying the regex to a Pandas Series using a list comprehension is the best way to go:

In [29]: s = pd.Series(                                     
    ['white male',
     'white male, white female',
     'hispanic male, 2 hispanic females',
     'black male, 2 white females'])

In [30]: %timeit s.str.replace('(?:two|2) (?P<race>[a-z]+) (?P<gender>(?:fe)?male)s', r'\g<race> \g<gender>, \g<race> \g<gender>')
1000 loops, best of 3: 205 µs per loop

In [31]: %timeit s.apply(lambda x: mult.sub(lambda x: x.group(1) + ' ' + x.group(1), x))
10000 loops, best of 3: 148 µs per loop

In [32]: %timeit [mult.sub(lambda x: x.group(1) + ' ' + x.group(1), i) for i in s]
100000 loops, best of 3: 14.6 µs per loop

Answer 4

The most simple way:

import pandas as pd

lst = ['Geeks', 'For', 'Geeks', 'is', 'portal', 'for', 'Geeks']
lst2 = [11, 22, 33, 44, 55, 66, 77]

df = pd.DataFrame(list(zip(lst, lst2)), columns =['Name', 'val'])

# \1 $1 \g<1>
df.replace(regex=r'(\w)(?P<ewe>\w)', value='\g<1>_\g<ewe>=')

## Output
           Name  val
0     G_e=e_k=s   11
1         F_o=r   22
2     G_e=e_k=s   33
3          i_s=   44
4  p_o=r_t=a_l=   55
5         f_o=r   66
6     G_e=e_k=s   77