如何在文本文件中查找一行中单词的频率 - Pyspark

Question

I have managed to make an RDD (in Pyspark) that looks like this:

[('This', (0, 1)), ('is', (0, 1)), ('the', (0, 1)), ('100th', (0, 1))...]

I used the following code: RDD=sc.textFile(_filepath_)

test1 = RDD.zipWithIndex().flatMap(lambda x: ((i,(x[1],1)) for i in x[0].split(" ")))

Practically, [(word, (line, freq)] so the above words are from the 1st line in the file (hence the 0) and freq is 1 for all words in the text, and I want it to count the times this word appears on this specific line, for the entire RDD. I thought of .reduceByKey(lambda x, y: x + y) but when I execute an action like .take(5) after that, it freezes (Ubuntu terminal - Oracle VirtualBox with plenty of RAM/disk space, if it helps).

Completely stuck on this stage.

What I need is basically, if the word 'This' is in first line and it's there 7 times, then the result will be [('This', (0, 7)), ...]

Answer 1

Solved it, but the answer may not be optimal.

RDD = sc.textFile(_filepath_) 
test1 = RDD.zipWithIndex().flatMap(lambda x: ((i,(x[1],1)) for i in x[0].split(" "))) 
test2 = test1.map(lambda x: ((x[0], x[1][0]), x[1][1])).reduceByKey(lambda x, y: x + y) 
Result_RDD = test2.map(lambda x: (x[0][0], (x[0][1], x[1])))

Answer 2

Given you have a list with all your lines, something like this should work for you:

#lines = [List of all your lines]
wordInLineFrequencyList = []

for i in range(len(lines)):
    currentLineWords = []
    for word in lines[i].split():
        if word not in currentLineWords:
            currentLineWords.append(word)
            wordInLineFrequencyList.append((word, (i, lines[i].count(word))))