文本预处理错误：['Errno 21]是目录

Question

I am trying to get all files from my directory and then run them through a series of def functions (python 3) and outputting each processed file into a certain directory. Below is my code:

   import re 
import glob
import sys
import string

#Create Stop_word Corpora
file1=open("/home/file/corps/stopwords.txt", 'rt', encoding='latin-1')
line= file1.read()
theWords=line.split()
stop_words=sorted(set(theWords)) # Stop Word Corpora

#Gather txt files to be processed
folder_path = "/home/file"
file_pattern = "/*txt"
folder_contents = glob.glob(folder_path + file_pattern)

#Read in the Txt Files
for file in folder_contents:
    print("Checking", file)
words= []
for file in folder_contents:
    read_file = open(file, 'rt', encoding='latin-1').read()
    words.extend(read_file.split())

def to_lowercase(words):
#"""Convert all characters to lowercase from list of tokenized words"""
    new_words=[]
    for word in words:
        new_word=word.lower()
        new_words.append(new_word)
    return new_words
def remove_punctuation(words):
#"""Remove punctuation from list of tokenized words"""
    new_words=[]
    for word in words:
        new_word = re.sub(r'[^\w\s]', '', word)
        if new_word != '':
            new_words.append(new_word)
    return new_words
def replace_numbers(words):
#""""""Replace all interger occurrences in list of tokenized words with textual representation"
    new_words=[]
    for word in words:
        new_word= re.sub(" \d+", " ", word)
    if new_word !='':
        new_words.append(new_word)
    return new_words

def remove_stopwords(words):
#"""Remove stop words from list of tokenized words"""
    new_words=[]
    for word in words:
        if not word in stop_words:
            new_words.append(word)
    return new_words
def normalize(words):
    words = to_lowercase(words)

    words = remove_punctuation(words)

    words = replace_numbers(words)

    words = remove_stopwords(words)
    return words

words = normalize(words)

# Write the new procssed file to a different location
append_file=open("/home/file/Processed_Files",'a')
append_file.write("\n".join(words))

This is the error I keep receiving:

I want the new text files to be sent to the directory above, after they have been ran through the def functions. So there should be 5 new files in the Processed_files directory above.

Answer 1

The traceback you present doesn't agree with the error reported in your question title.

But your code does this twice:

for word in words:
    new_word = re.sub(r'[^\w\s]', '', word)
if new_word != '':
    new_words.append(new_word)

If words is empty, then the for word in words loop never gets executed, even once. And if it doesn't get executed even once then no value ever gets assigned to new_word . So, in that case, when your code does if new_word != '': you will get the error new_word referenced before assignment . That is because your code is asking what is in new_word but it is unassigned.

This problem will go away if you code it like this:

for word in words:
    new_word = re.sub(r'[^\w\s]', '', word)
    if new_word != '':
        new_words.append(new_word)

which I suspect is what you meant, anyway.

Answer 2

I would suggest 3 changes:

1. Create an empty list and add all words to it

    words = [] for file in folder_contents: read_file = open(file, 'rt', encoding='latin-1').read() words.extend(read_file.split()) 

2. Correctly convert a list into a str

    append_file.write("\\n".join(words))) 

3. Fix incorrect indentation

    words = normalize(words) 

   and

    for word in words: new_word = re.sub(r'[^\\w\\s]', '', word) if new_word != '': new_words.append(new_word)