将熊猫中的句子分成句子数和单词

Question

I have a pandas dataframe like this:

Text            start    end    entity     value
I love apple      7       11    fruit      apple
I ate potato      6       11    vegetable  potato

I have tried to use a for loop It's running slow and I don't think this is what we should do with pandas.

I want to create another pandas dataframe base on this like:

Sentence#         Word        Tag
  1                I         Object 
  1               love       Object
  1               apple      fruit
  2                I         Object
  2               ate        Object
  2               potato     vegetable

Split the text column into words and sentence numbers. Other than the entity word, the other words will be tagged as Object.

Answer 1

Use split , stack and map :

u = df.Text.str.split(expand=True).stack()

pd.DataFrame({
    'Sentence': u.index.get_level_values(0) + 1, 
    'Word': u.values, 
    'Entity': u.map(dict(zip(df.value, df.entity))).fillna('Object').values
})

   Sentence    Word     Entity
0         1       I     Object
1         1    love     Object
2         1   apple      fruit
3         2       I     Object
4         2     ate     Object
5         2  potato  vegetable

Side note: If running v0.24 or later, please use .to_numpy() instead of .values .

Answer 2

I am using unnesting here after str.split

df.Text=df.Text.str.split(' ')
yourdf=unnesting(df,['Text'])
yourdf.loc[yourdf.Text.values!=yourdf.value.values,'entity']='object'
yourdf
     Text  start  end     entity   value
0       I      7   11     object   apple
0    love      7   11     object   apple
0   apple      7   11      fruit   apple
1       I      6   11     object  potato
1     ate      6   11     object  potato
1  potato      6   11  vegetable  potato

Answer 3

Using the expand function I posted in this thread , you can

df = expand(df, 'Text', sep=' ')

Then simple

df['Tag'] = np.where(df.Text.ne(df.value), ['Object'], df.entity)


>>> df[['Text', 'Tag']]

    Text    Tag
0   I       Object
1   love    Object
2   apple   fruit
3   I       Object
4   ate     Object
5   potato  vegetable

---

def expand(df, col, sep=','):
    r = df[col].str.split(sep)
    d = {c: df[c].values.repeat(r.str.len(), axis=0) for c in df.columns}
    d[col] = [i for sub in r for i in sub]
    return pd.DataFrame(d)