如何在Python列表中将单词反令牌化回原始形式

Question

libOfSentences = ["Get help with the display",
                 "Display is not working properly", "I need some help"]
#removing stopwords

for i in libOfSentences:
     sentence = word_tokenize(j) #tokenize each individual word
     sentence = filter(lambda x: x not in string.punctuation, sentence) 
     cleaned_text = filter(lambda x: x not in stop_words, sentence) 

     removedStopwordsList = " ".join(cleaned_text) 

removedStopwordsList has now joined the sentences back together but I want to keep it in a list. The desired output is like this:

["Get help display", "Display not working properly", "I need some help"]

I want to have removedStopwordsList still be a list I can loop through for example

removedStopwordsList[0] 

gives me

"G D I" 

right now but I want removedStopwordsList[0]

to output

"Get help display"

The join function is what is stopping this from occurring right now but I can't find a better workaround.

Answer 1

I want to have removedStopwordsList still be a list

Then just make it a list instead of making it a string:

removedStopwordsList = list(cleaned_text)

Although you can do this even more simply by using a list comprehension instead of calling filter :

removedStopwordsList = [x for x in sentence if x not in stop_words]

map and filter are great when you have a function you want to call on each element, but when you have an arbitrary expression, which you have to wrap up in lambda to turn into a function call, it's simpler and more readable to just use a list comprehension or generator expression.

And you can similarly simplify the previous line. So:

for i in libOfSentences:
    sentence = word_tokenize(j) #tokenize each individual word
    sentence = (x for x in sentence if x not in string.punctuation)
    removedStopwordsList = [x for x in sentence if x not in stop_words]

---

If you need to have the joined-up string around as well, that's fine; you can have a second variable:

removedStopwordsString = " ".join(removedStopwordsList)

If you really want a single object that can behave both ways, it wouldn't be hard to write such a class, but it would just be ugly. And under the covers, it's just going to have a self.list_of_words and self.joined_string that it delegates to anyway. So, what would be the point?

At any rate, I doubt you need to keep the string around. If you ever want to print it out, you can just join it on the fly:

print(" ".join(removedStopwordsList))

… or even expand it into separate printables:

print(*removeStopwordsList)

---

If you're trying to gather all of those lists into one big list, you have to actually write code to do that. Obviously if you do removeStopwordsList = <anything> each time through the loop, you're just replacing it each time through. You need to append that to some bigger list if you want to keep all those lists around. For example:

listOfLists = []
for i in libOfSentences:
    sentence = word_tokenize(j) #tokenize each individual word
    sentence = (x for x in sentence if x not in string.punctuation)
    removedStopwordsList = [x for x in sentence if x not in stop_words]
    listOfLists.append(removedStopwordsList)

And now, if you print out listOfLists , it'll be a list of two lists of words; listOfLists[0] will be the first list; listOfLists[0][0] will b the first word of the first list; etc.