在Apache Spark中对RDD进行分区，使得一个分区包含在一个文件中

Question

I am creating a single RDD of 2.csv files like this

val combineRDD = sc.textFile("D://release//CSVFilesParellel//*.csv")

Then I want to define custom partition on this RDD such that one partition must contain one file. so that each partition ieone csv file is processed across one node for faster data processing

Is it possible to write a custom partitioner based on the file size or the number of lines in one file or the end of file character of one file ?

How do I achieve this ?

The structure of one file looks something like this:

00-00

Time(in secs) Measure1 Measure2 Measure3..... Measuren

0

0.25

0.50

0.75

1

...

3600

---

1.The first row of data contains the hours: mins Each file contains data for 1 hour or 3600secs

2.The first column is a second divided into 4 parts of 250 ms each and data recorded for 250 ms

3. For every file I want to add the number of hours: mins to the seconds so that my time looks something like this hours-mins-secs. But the catch is I dont want this process to happen sequentially

4. I am using the for-each function for getting each file name -> then creating an RDD of the data in the file and adding the time as specified above.

5. But what I want is that every file should go to one node for processing and calculating time as opposed to data in one file getting distributed across nodes for calculating the time.

Thank you.

Regards,

Vinay Joglekar

Answer 1

Lets go back to basics.

1. Philosphy of BigData move Process to the data and not data to process. this way one increases parallelism and hence I/O throughput
2. One partitioner taking one file will decrease the parallelism not increase.
3. Simplest way to achieve this is use textInpuTFormat and have your input files compresses by gzip or lzo( no lzo indexing should be done).

4. Gzip being non splittable will force one file going to one partition but this in no way will help in anyKind throughput increase

5. To Write Custom Input Format Extend from FileInputFormat and give your splitlogic and recordReader logic.

To use custom input format in spark please follow

http://bytepadding.com/big-data/spark/combineparquetfileinputformat/

Answer 2

Simple answer without questioning why you're doing this. Load the files separately so you know the filename being loaded

// create firstRDD containing a new attribute `filename=first.csv`
val firstRDD = sc.textFile("D://release//CSVFilesParellel//first.csv")
    .map(line => new CsvRecord(line))

// create secondRDD containing a new attribute `filename=second.csv`
val secondRDD = sc.textFile("D://release//CSVFilesParellel//second.csv")
    .map(line => new CsvRecord(line))

// now create a pair RDD and re-partition on the filename
val partitionRDD = firstRDD.union(secondRDD)
    .map(csvRecord => (csvRecord.filename,csvRecord))
    .partitionBy(customFilenamePartitioner)

The following quote from here

To implement your customFilenamePartitioner, you need to subclass the org.apache.spark.Partitioner class and implement three types of methods :

NumPartitions : Int, which returns the number of partitions that you will create it as per the requirements.

getPartition(key: Any) : Int, which returns the partition ID ranging from (0 to numPartitions-1) for a given key.

equals(), : the standard Java programming equality method. This is important to implement because Spark application will need to test your Partitioner object against other instances by its own terms when it decides whether two of your RDDs are partitioned the same way as it is required.

Keep in mind that a repartition will most likely trigger an expensive shuffle, so unless you are going to be repeatedly querying this newly partitioned RDD you may be better off solving your problem another way.