Spark和Python使用自定义文件格式/生成器作为RDD的输入

Question

I would like to ask about input possibilities in Spark. I can see from http://spark.apache.org/docs/latest/programming-guide.html , that I can use sc.textFile() for reading the text files to RDD, but I would like to do some preprocessing, before the distribution to RDD will happen, for example my file might be in JSON format eg. {id:123, text:"...", value:6} and I would like to use only certain fields of the JSON for further processing.

My idea was if there is possible somehow to use Python generator as an input to the SparkContext?

Or if there is some more natural way in Spark how to process custom, not text only files by Spark?

EDIT:

It seems that accepted answer should work, but it has moved me to my more practical following question Spark and Python trying to parse wikipedia using gensim

Answer 1

The fastest way to do this is probably to load the text file as-is and do your processing to select desired fields on the resulting RDD. This parallelizes that work across the cluster and will scale more efficiently than doing any preprocessing on a single machine.

For JSON (or even XML), I don't think you need a custom input format. Since PySpark executes within a Python environment, you can use functions regularly available to you in Python to deserialize the JSON and extract the fields you want.

For example:

import json

raw = sc.textFile("/path/to/file.json")
deserialized = raw.map(lambda x: json.loads(x))
desired_fields = deserialized.map(lambda x: x['key1'])

desired_fields is now an RDD of all the values under key1 in the original JSON file.

You can use this pattern to extract a combination of fields, split them by whitespace, or whatever.

desired_fields = deserialized.map(lambda x: (x['key1'] + x['key2']).split(' '))

And if this gets too complicated, you can replace the lambda with a regular Python function that does all the preprocessing you want and just call deserialized.map(my_preprocessing_func) .

Answer 2

Yes, you can create an RDD from a python variable using SparkContext.parallelize() :

data = [1, 2, 3, 4, 5]
distData = sc.parallelize(data)
distData.count()   # 5

This variable can also be an iterator.