找出列表中三个单词在给定文档中共同出现的频率

Question

I have multiple.txt files in desktop loaded as a data frame in python. I am working in python in the data frame, where 'text' is the name of the column that is my interest. Column 'text' consists of multiple.txt documents.

I also have three lists of words in mind: these are:

credit=['borrow', 'lend'],
policy=['Fed', 'fund rate','zero'],
trade=['deficit', 'surplus'],

My goal is to construct the index that measures the frequency of any of the words from the three lists in combination in a given sentence in the text file by applying it for each document separately. For example if 'borrow', 'fund' and 'surplus' co-occurred in a given sentence, it will enter code here be counted as 1.

I know how to do it to count using a single word as follows:

my_dir_path ='C:/Users/desktop'
results = defaultdict(list)
for file in Path(my_dir_path).iterdir():
    with open(file, "r") as file_open:
        results["file_name"],(file.name)
        results["text"].append(file_open.read())
        df = pd.DataFrame(results)

to get the frequencty of the word policy across documents I used this code:

df['policy']=df['text'].apply(lambda x: len([word for word in x.split() if     word=='policy']))

How can I do it in python? Thanks in advance for any help?

Answer 1

I'd be tempted to use regular expressions for doing the matching of words within sentences, using lookahead/behind we could use something like:

(?<!\w)borrow(?!\w)

and would find "borrow" in "can I borrow that" and "will borrow." but not "borrowing" . I'm unsure what you actually want to do here, but I'd suggest learning how to use regular expressions as they would allow you to express these options easily

to make the following code shorter, I define a function to compile a "word" into a regex object:

import re

def matcher(word):
    return re.compile(fr'(?<!\w){word}(?!\w)', re.IGNORECASE)

re_credit = [
    matcher('borrow'),
    matcher('fund'),
]

next I write a function to split a string up into sentences so we can count co-occurances of words:

from nltk.tokenize import sent_tokenize

def count_sentences_matching_words(text, regexes):
    count = 0
    for sentence in sent_tokenize(text):
        if all(reg.search(sentence) for reg in regexes):
            count += 1
    return count

next we can test it with some text:

para = "My goal is to construct the index that measures the frequency of any of the words from the three lists in combination in a given sentence in the text file by applying it for each document separately. For example if 'borrow', 'fund' and 'surplus' co-occurred in a given sentence, it willenter code here be counted as 1."

count_sentences_matching_words(para, re_credit)

if you wanted to use this with pandas you could do the obvious:

df['credit'] = df['text'].apply(lambda x: count_sentences_matching_words(x, re_credit))

it's probably worth rearranging this code, eg just doing the sentence tokenization once per file. but it would depend on more details than you've shared