熊貓：檢查數據框的每個單元格中是否存在值

Question

我正在使用python 2.7，並希望根據每個單元格中列表的每個值的存在來創建一列。

這是一個數據示例：

|    query      |
-----------------
| handbag woman |
| shoe man      |
| t-shirt baby  |
| watch unisex  |
| dress         |

我有一個要檢查的值列表：

gender_list=['woman', 'man', 'baby', 'unisex']

我期望的結果：

|   query      |   gender
-----------------------
|   handbag    |  woman 
|   shoe       |  man      
|   t-shirt    |  baby  
|   watch      |  unisex  
|   dress      |  None

這是我已經嘗試過的：

for gender in gender_list:
    df['gender']=df['query'].map(lambda x : gender if (x.find(gender) != -1) else None)
    df['query']=df['query'].map(lambda x : x.replace(gender, '').strip() if (x.find(gender) != -1) else x)

Answer 1

首先，在大熊貓中，最好不要使用循環，因為速度慢（適用於引擎蓋下的循環），而是使用矢量化解決方案。

使用extract和正則表達式replace用|連接所有值| 並使用word boundary進行完全匹配：

gender_list=['woman', 'man', 'baby', 'unisex']
#exact match is not important
#pat = '|'.join(gender_list)
pat = '|'.join(r"\b{}\b".format(x) for x in gender_list)
print (pat)
\bwoman\b|\bman\b|\bbaby\b|\bunisex\b

df['gender'] = df['query'].str.extract('('+ pat + ')', expand=False)
df['query'] = df['query'].str.replace(pat, '').str.strip()
print (df)
     query  gender
0  handbag   woman
1     shoe     man
2  t-shirt    baby
3    watch  unisex
4    dress     NaN

差異：

print (df)
           query
0  handbag woman
1      shoe many <-man change to many
2   t-shirt baby
3   watch unisex
4          dress

gender_list=['woman', 'man', 'baby', 'unisex']
pat = '|'.join(r"\b{}\b".format(x) for x in gender_list)
df['gender'] = df['query'].str.extract('('+ pat + ')', expand=False)
df['query'] = df['query'].str.replace(pat, '').str.strip()
print (df)
       query  gender
0    handbag   woman
1  shoe many     NaN <-many not extracted
2    t-shirt    baby
3      watch  unisex
4      dress     NaN

---

gender_list=['woman', 'man', 'baby', 'unisex']
pat = '|'.join(gender_list)
df['gender'] = df['query'].str.extract('('+ pat + ')', expand=False)
df['query'] = df['query'].str.replace(pat, '').str.strip()
print (df)
     query  gender
0  handbag   woman
1   shoe y     man <-stay y from many
2  t-shirt    baby
3    watch  unisex
4    dress     NaN

時間 ：

df = pd.DataFrame({'query': ['handbag woman', 'shoe man', 't-shirt baby', 'watch unisex', 'dress', 'manpower']})
print (df)

df = pd.concat([df] * 10000, ignore_index=True)


In [299]: %%timeit
     ...: pat = '|'.join(r"\b{}\b".format(x) for x in gender_list)
     ...: df['gender'] = df['query'].str.extract('('+ pat + ')', expand=False)
     ...: df['query'] = df['query'].str.replace(pat, '').str.strip()
     ...: 
     ...: 
1 loop, best of 3: 143 ms per loop

In [300]: %%timeit
     ...: gender_set = set(gender_list)
     ...: 
     ...: def gender_sep(row):
     ...:     lst = row['query'].split(' ')
     ...:     gender = next(iter(gender_set & set(lst)), None)
     ...:     return (' '.join(lst), None) if not gender else \
     ...:            (' '.join(i for i in lst if i!= gender), gender)
     ...: 
     ...: df['query'], df['gender'] = list(zip(*df.apply(gender_sep, axis=1)))
     ...: 
1 loop, best of 3: 933 ms per loop

編輯：

對於更常見的通用解決方案，需要通過re.escape轉義正則表達式值：

import re

gender_list=['woman', 'man', 'baby', 'girl & boy']
pat = '|'.join(r"\b{}\b".format(re.escape(x)) for x in gender_list)
df['gender'] = df['query'].str.extract('('+ pat + ')', expand=False)
df['query'] = df['query'].str.replace(pat, '').str.strip()

Answer 2

這是一種方式。 它不是最有效的，但可讀性強，易於調整/維護。

import pandas as pd

df = pd.DataFrame({'query': ['handbag woman', 'shoe man', 't-shirt baby', 'watch unisex', 'dress', 'manpower']})

gender_list = ['woman', 'man', 'baby', 'unisex']
gender_set = set(gender_list)

def gender_sep(row):
    lst = row['query'].split(' ')
    gender = next(iter(gender_set & set(lst)), None)
    return (' '.join(lst), None) if not gender else \
           (' '.join(i for i in lst if i!= gender), gender)

df['query'], df['gender'] = list(zip(*df.apply(gender_sep, axis=1)))

#       query  gender
# 0   handbag   woman
# 1      shoe     man
# 2   t-shirt    baby
# 3     watch  unisex
# 4     dress    None
# 5  manpower    None