从Spark中的单行生成多行

Question

I have some data in cassandra with the following data model:

transaction_id : uuid
start_date: timestamp
end_date: timestamp
PRIMARY KEY(transaction_id)

Now I want to transform this data into something :

aggregation_date : timestamp
number_of_active_transaction_0 : int
number_of_active_transaction_1 : int
number_of_active_transaction_2 : int
...
number_of_open_transaction_23 : int
PRIMARY KEY((aggregation_date))

Currently I have created a function which takes the start and end dates and returns a tuple of transaction_date (just the Date part) and 24 size array with value 1 for the hours when the transaction was active and map the original RDD into a PairRDD with transaction_date (just the Date part) as key and the array as value. After this performing a reduce on the key and adding all the individual elements of the array to get the desired output.

Now the problem is, there are instances when the transaction starts late in the night and is completed post midnight so in such cases I want to return 2 rows from my function so that for every transaction I get 2 rows in the returned RDD.

Spark version: 1.2.2
API used is Scala
Spark Cassandra connector version 1.2.2

Answer 1

You will likely want to use flatMap , with flatMap you can output multiple (including zero) elements for each input.

However you also mention you are performing a reduce on the key, if its during this phase when you need to output multiple elements you can just produce a list during the reduceByKey and then just do an identity flatMap , which will flatten all the results.