修改Spark读取的文本文件

Question

I am trying to count words in few text files in a Hadoop Cluster while using Spark. I manage to get the word count but I also want to do some further modifications such as ignoring numbers or transforming all words to lower case. I can't iterate over the RDD-data normally. I've tried using collect() but the map function does not accept list as an argument. I've also tried to apply regex logic directly to the "filter" function of RDD but found no success. This is the code I've come up this far, it works without the parts that I've commented out.

from pyspark import SparkConf, SparkContext
import re
conf = SparkConf().setAppName("Word count")
sc = SparkContext(conf=conf)
sc.setLogLevel("WARN")
text = sc.textFile("/data/book/*.txt") \
       .flatMap(lambda line: line.split())

#handledText = text.map(lambda s: s.replace("\d", "", text))
counts = text.map(lambda word: (word, 1)) \
         .groupByKey() \
         .map(lambda p: (p[0], sum(p[1])))
res = counts.takeOrdered(text.count(), key=lambda p: -p[1])
print(res)

Answer 1

text.map(lambda s: s.replace("\\d", "", text))

You are confusing map() built-in function in Python with DataFrame.map() of Spark... No, the text parameter is not valid there.

Try this

def lower_no_digit(word):
    return lower(word.replace(r'\d+', ''))

counts = text.map(lower_no_digit) \ 
             .filter(lambda w : len(w) > 0) \
             .map(lambda word: (word, 1)) \

Which maps a function over the words and filters out the empty ones before applying (word, 1)

Aside - Doing the same in SparkSQL is somewhat simpler and doesn't require manually putting (word, 1)

I've tried using collect()

Do not map(lambda x : ... , df.collect()) . That will bring all the data to the local Spark driver, and defeats the purpose of running a distributed processing framework.