删除 Python 上的停用词和其他任务

Question

So, I have been given a .txt file (name:newtext) containing a novel and a .txt file (name:stopwords) containing a list of stopwords and I have to work on these two (without importing any other processing tools such as NLTK etc.) and I need to perform these tasks:

• identify words
• remove stop words using the list of “stop words”
• determine frequencies of occurrences for each word after the stop-word removal
• Print out the top ten most frequent ones, together with their frequency counts

I am really lost.

Answer 1

I'll give you some hints:

1. When you have opened your txt file with the text, you can iterate over the text word for word.
2. You can store your stop words which you also have read from the stop_word.txt file in a list.
3. During the iteration of words, if the word you are currently looking at is not a stop word, save this word to a new string. (This removes all stop words).
4. After you have created your new string without stopwords (dont forget to add a space between words your adding to the new string) you can do like this to split the new string and count all occurances.

List item

for word in new_words.split(" "):
    if not word_count.get(word, False):
        word_count[word] = 1
    else:
        word_count[word] += 1

for word in word_count.keys():
    print(f"Number of occurances of {word} was {word_count[word]}.")

Answer 2

Any how, I thought of adding an answer for this. Its a bare logic, which I did not test. Hopefully it will become handy!

novel = open("newtext.txt", "r").read()
s_words = open("stopwords.txt", "r").readlines()
s_words = [x.strip() for x in s_words]

# identify all words in the novel
all_words = novel.split(" ")

# remove stop words using the list of “stop words”
no_stop_words = [x for x in all_words if x not in s_words]

# determine frequencies of occurrences for each word after the stop-word removal
frequencies  = {word: no_stop_words.count(word) for word in no_stop_words}

# Print out the top ten most frequent ones, together with their frequency counts
for word, frequency in sorted(frequencies.items(), key=lambda x: x[1], reverse=True)[:10]:
    print(word, frequency)