split sentences (strings) in pandas into separate rows of words with sentence numbering

Question

I have a pandas dataframe like this:

sn  sentence                    entity
1.  an apple is an example of?  an apple is example of fruit
2.  a potato is an example of?  a potato is example of vegetable

I want to create another pandas dataframe that looks like below: where the length of the sentence and entity are the same as below

Sentence#   Word    Entity
  1         An      an 
  1         apple   apple
  1         is      is
  1         an      example
  1         example of 
  1         of?     fruit
  2         A       a 
  2         potato  potato
  2         is      is
  2         an      example
  2         example of
  2         of?     vegetable

What I have tried so far

df = data.sentence.str.split(expand=True).stack()

pd.DataFrame({
    'Sentence': df.index.get_level_values(0) + 1, 
    'Word': df.values, 
    'Entity': 
})

The last bit on "Entity" is what I can't seem to get right

I also tried to split and stack the entity column, like so?

df2 = data.sentence.str.split(expand=True).stack() 

and then attempt to put all back together

pd.DataFrame({
    'Sentence': df.index.get_level_values(0) + 1, 
    'Word': df.values, 
    'Entity': df2.values
})

but then I get ValueError: arrays are must all be of the same length

len(df) = 536810, len(df2) = 536802

I am new to python. Any help or pointers appreciated.

Answer 1

Let us try str.split then do explode and concat back

s=df.set_index('sn')
s=pd.concat([s[x].str.split(' ').explode() for x in s.columns],axis=1).reset_index()
s
Out[79]: 
    sn sentence     entity
0    1       an         an
1    1    apple      apple
2    1       is         is
3    1       an    example
4    1  example         of
5    1      of?      fruit
6    2        a          a
7    2   potato     potato
8    2       is         is
9    2       an    example
10   2  example         of
11   2      of?  vegetable

Answer 2

Here is a simple way to do it without explicit iteration -

1. Set sn to index
2. Applymap string split to each of the cells in dataframe
3. Explode the lists over axis 0
4. Reset index

df.set_index('sn').\
applymap(str.split).\
apply(pd.Series.explode, axis=0).\
reset_index()

    sn sentence     entity
0    1       an         an
1    1    apple      apple
2    1       is         is
3    1       an    example
4    1  example         of
5    1      of?      fruit
6    2        a          a
7    2   potato     potato
8    2       is         is
9    2       an    example
10   2  example         of
11   2      of?  vegetable

Answer 3

One approach without loops

new_df = (df.set_index('sn')
            .stack()
            .str.split(expand=True)
            .stack()
            .unstack(level=1)
            .reset_index(level=0, drop=0)
                        )
print(new_df)

Output

    sn sentence     entity
0  1.0       an         an
1  1.0    apple      apple
2  1.0       is         is
3  1.0       an    example
4  1.0  example         of
5  1.0      of?      fruit
0  2.0        a          a
1  2.0   potato     potato
2  2.0       is         is
3  2.0       an    example
4  2.0  example         of
5  2.0      of?  vegetable