从python列表中删除单词？

Question

I am a complete noob in python and web scraping and have ran into some issues quite early. I have been able to scrape a Dutch news website their titles and splitting the words. Now my objective is to remove certain words from the results. For instance I don't want word like "het" and "om" in the list. Does anyone know how I can do this? (I'm using python requests and BeautifulSoup)

 import requests from bs4 import BeautifulSoup url="http://www.nu.nl" r=requests.get(url) soup=BeautifulSoup(r.content) g_data=soup.find_all("span" , {"class": "title"}) for item in g_data: print item.text.split() 

Answer 1

In natural language processing, the term for excluding common words is called "stop words".

Do you want to preserve the order and count of each word, or do you just want the set of words that appear on the page?

If you just want the set of words that appear on the page, using sets is probably the way to go. Something like the following might work:

# It's probably more common to define your STOP_WORDS in a file and then read
# them into your data structure to keep things simple for large numbers of those
# words.
STOP_WORDS = set([
    'het',
    'om'
])

all_words = set()
for item in g_data:
    all_words |= set(item.text.split())
all_words -= STOP_WORDS
print all_words

If, on the other hand, you care about the order, you could just refrain from adding stop words to your list.

words_in_order = []
for item in g_data:
    words_from_span = item.text.split()
    # You might want to break this out into its own function for modularity.
    for word in words_from_span:
        if word not in STOP_WORDS:
            words_in_order.append(word)
print words_in_order

If you don't care about order but you want frequency, you could create a dict (or defaultdict for convenience) of words to counts.

from collections import defaultdict
word_counts = defaultdict(int)
for item in g_data:
    # You might want to break this out into its own function for modularity.
    for word in item.text.split():
        if word not in STOP_WORDS:
            word_counts[word] += 1
for word, count in word_counts.iteritems():
    print '%s: %d' % (word, count)