将3D单词列表展平为2D

Question

I have a pandas column with text strings. For simplicity ,lets assume I have a column with two strings.

s=["How are you. Don't wait for me",  "this is all fine"]

I want to get something like this:

[["How", "are","you"],["Don't", "wait", "for", "me"],["this","is","all","fine"]]

Basically take each sentence of a document and tokenism into list of words. So finally I need list of list of string.

I tried using a map like below:

nlp=spacy.load('en')
def text_to_words(x):
    """ This function converts sentences in a text to a list of words

    """

    global log_txt
    x=re.sub("\s\s+" , " ", x.strip())
    txt_to_words= [str(doc).replace(".","").split(" ") for doc in nlp(x).sents]

    #log_txt=log_txt.extend(txt_to_words)

    return txt_to_words

The nlp from spacy is used to split a string of text into list of sentences.

log_txt=list(map(text_to_words,s))

log_txt

But this as you know would put all of the result from both the documents into another list

[[['How', 'are', 'you'], ["Don't", 'wait', 'for', 'me']],
 [['this', 'is', 'all', 'fine']]]

Answer 1

You'll need a nested list comprehension. Additionally, you can get rid of punctuation using re.sub .

import re

data = ["How are you. Don't wait for me",  "this is all fine"]
words = [
    re.sub([^a-z\s], '', j.lower()).split() for i in data for j in nlp(i).sents
]

Or,

words = []
for i in data:
    ... # do something here
    for j in nlp(i).sents:
        words.append(re.sub([^a-z\s], '', j.lower()).split())

Answer 2

There is a much simpler way for list comprehension. You can first join the strings with a period '.' and split them again.

[x.split() for x in '.'.join(s).split('.')]

It will give the desired result.

[["How", "are","you"],["Don't", "wait", "for", "me"],["this","is","all","fine"]]

For Pandas dataframes, you may get an object, and hence a list of lists after tolist function in return. Just extract the first element.

For example,

import pandas as pd

def splitwords(s):
    s1 = [x.split() for x in '.'.join(s).split('.')]
    return s1

df = pd.DataFrame(s)
result = df.apply(splitwords).tolist()[0]

Again, it will give you the preferred result.

Hope it helps ;)