删除停用词和标点符号

Question

I parse information from a news website. Each news is a dictionary that is stored inside translated_news variable. Each news has its title, url and country. Then I try to iterate over each news title and delete stop-words and punctuation signs. I've written this code:

for new in translated_news:
    tk = tokenize(new['title'])
    # delete punctuation signs & stop-words
    for t in tk:
        if (t in punkts) or (t+'\n' in stops):
            tk.remove(t)
tokens.append(tk)

Tokenize is a function that returns a list of tokens. Here's an example of the output:

['medium', ':', 'russian', 'athlete', 'will', 'be', 'admit', 'to', 'the', '2018', 'olympics', 'in', 'neutral', 'status']

Here's the same output, but with deleted stop-words and punctuation:

['medium', 'russian', 'athlete', 'be', 'admit', 'the', 'olympics', 'neutral', 'status']

The problem is: even though the words 'the' and 'be' are included in my stop-words list, they were not deleted from the news title. However, on other titles it sometimes works correctly:

['wada', 'acknowledge', 'the', 'reliable', 'information', 'provide', 'to', 'rodchenkov'] ['wada', 'acknowledge', 'reliable', 'information', 'provide', 'rodchenkov']

Here 'the' was deleted from the title. I don't understand what is wrong with the code and why sometimes the output is perfect and sometimes not.

Answer 1

You have to iterate on tokenize(new['title']) and use De Morgan's laws to simplify the if statement:

import string

stops = ['will', 'be', 'to', 'the', 'in']

tk = ['medium', ':', 'russian', 'athlete', 'will', 'be', 'admit', 'to', 'the',
      '2018', 'olympics', 'in', 'neutral', 'status']

# delete punctuation signs & stop-words
tk = []
for t in tokenize(new['title']):
    # if not ((t in string.punctuation) or (t in stops)):
    if (t not in string.punctuation) and (t not in stops): # De Morgan's laws
        tk.append(t)
print(tk)

will print:

['medium', 'russian', 'athlete', 'admit', '2018', 'olympics', 'neutral', 'status']

You can get rid of new lines in stops words:

stops = ['will\n', 'be\n', 'to\n', 'the\n', 'in\n']
stops = [item.strip() for item in stops]
print(stops)

will print:

['will', 'be', 'to', 'the', 'in']

---

The solution suggested from incanus86 does work:

tk = [x for x in tokenize(new['title']) if x not in stops and x not in string.punctuation]

but you won't be asking in SO if you knew about list comprehensions .

---

I don't understand what is wrong with the code and why sometimes the output is perfect and sometimes not.

While iterating on tk items you do miss 'be' and 'the' because you are removing tk items as seen in code:

import string

stops = ['will', 'be', 'to', 'the', 'in']

tk = [
    'medium',  # 0
    ':',  # 1
    'russian',  # 2
    'athlete',  # 3
    'will',  # 4
    'be',  # 5
    'admit',  # 6
    'to',  # 7
    'the',  # 8
    '2018',  # 9
    'olympics',  # 10
    'in',  # 11
    'neutral',  # 12
    'status'  # 13
]

# delete punctuation signs & stop-words
for t in tk:
    print(len(tk), t, tk.index(t))
    if (t in string.punctuation) or (t in stops):
        tk.remove(t)

print(tk)

will print:

(14, 'medium', 0)
(14, ':', 1)
(13, 'athlete', 2)
(13, 'will', 3)
(12, 'admit', 4)
(12, 'to', 5)
(11, '2018', 6)
(11, 'olympics', 7)
(11, 'in', 8)
(10, 'status', 9)
['medium', 'russian', 'athlete', 'be', 'admit', 'the', '2018', 'olympics', 'neutral', 'status']

You do miss "russian" , "be" , "the" and "neutral" .
Index of "athlete" is 2 and index of "will" is 3 because you removed ":" from tk.
Index of "admit" is 4 and index if "to" is 5 because you removed "will" from tk.
Index of "2018" is 6, index of "olympics" is 7, index of "in" is 8 and index of "status" is 9.

You MUST not change list while iterating on it!

Answer 2

Try getting rid of the newline character.

something like this

tk = [x for x in tokenize(new['title']) if x not in stops and x not in string.punctuation]