使用python从文本文件中提取段落并排除目录和标题

Question

I download a paper from website and would like to use NLTK to do topic modeling with complete sentence. Therefore, I try to exclude irrelevant words or not complete sentences in text file. But, I still can't remove those single word.

For example, the format in text file

I only want to get the last sentence. and as the following code is to divide a text into a list of sentence.

tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
tokenizer.tokenize(data)
print('\n'.join(tokenizer.tokenize(data)))

But, how could I exclude those single words line by line? Thank you

Answer 1

This can be done by using the split method on each line of the text file.

file_list = []
file = open('Your Text File.txt', 'r')
for line in file:
    splitted_lines = line.split(' ')
    if len(splitted_lines) > 1 :
        file_list.append(' '.join(splitted_lines))

outfile = ''.join(file_list)
file_out = open('outfile.txt', 'w')
file_out.write(outfile)