分割时间序列数据框

Question

I have a dataframe where I have different parameters as columns and a timestamp for each row of parameters.

What I want to do is to split the dataframe into windows, where column values from each row get all appended into a single row. This will enable me to run clustering using these as features.

For example, I want to transform dataframe like this (window size 3):

2017-01-01 00:00:01, a1, b1, c1
2017-01-01 00:00:02, a2, b2, c2
2017-01-01 00:00:03, a3, b3, c3
2017-01-01 00:00:04, a4, b4, c4
2017-01-01 00:00:05, a5, b5, c5
2017-01-01 00:00:06, a6, b6, c6
2017-01-01 00:00:07, a7, b7, c7

Into something like this:

2017-01-01 00:00:01, 2017-01-01 00:00:03, a1, a2, a3, b1, b2, b3, c1, c2, c3
2017-01-01 00:00:04, 2017-01-01 00:00:06, a4, a5, a6, b4, b5, b6, c4, c5, c6

I need to preserve information which time interval belongs to which cluster, after clustering, so that is why I also need to keep the time ranges. The last instant in the example was excluded as there's not enough measurements to create another window.

How can I do this using Spark?

Answer 1

Let's start with some data, according to your description:

from pyspark.sql.functions import unix_timestamp

df = sc.parallelize([("2017-01-01 00:00:01", 2.0, 2.0, 2.0),
("2017-01-01 00:00:08", 9.0, 9.0, 9.0),
("2017-01-01 00:00:02", 3.0, 3.0, 3.0),
("2017-01-01 00:00:03", 4.0, 4.0, 4.0),
("2017-01-01 00:00:04", 5.0, 5.0, 5.0),
("2017-01-01 00:00:05", 6.0, 6.0, 6.0),
("2017-01-01 00:00:06", 7.0, 7.0, 7.0),
("2017-01-01 00:00:07", 8.0, 8.0, 8.0)]).toDF(["time","a","b","c"])
df = df.withColumn("time", unix_timestamp("time", "yyyy-MM-dd HH:mm:ss").cast("timestamp"))

> Spark 2.0

We could generate a new interval column using the ceil() function , by which then we can group your data and collect all the other variables into one flat list.

To guarantee correct ordering inside the resulting lists, irrespective of initial order, we'll use Window functions, to partition your data by date , creating a rank column ordered by the time .

from pyspark.sql.window import Window
from pyspark.sql.functions import ceil

df = df.withColumn("date", df["time"].cast("date")) \
       .withColumn("interval", ((ceil(df["time"].cast("long") / 3L))*3.0).cast("timestamp")) 

window = Window.partitionBy(df['date']).orderBy(df['time'])

Because we'll collect the rank column into the nested list for correct ordering, we'll define an udf that eventually unpacks all values in the nested lists, but the first one , which is the rank :

def unnest(col):

  l = [item[1:] for item in col]
  res = [item for sublist in l for item in sublist]

  return(res)

unnest_udf = udf(unnest)

Now we putting everything together:

from pyspark.sql.functions import rank
from pyspark.sql.functions import collect_list, array

df.select('*', rank().over(window).alias('rank')) \
  .groupBy("interval") \
  .agg(collect_list(array("rank","a", "b","c")).alias("vals")) \
  .withColumn("vals", unnest_udf("vals")) \
  .sort("interval") \
  .show(truncate = False)
+---------------------+---------------------------------------------+
|interval             |vals                                         |
+---------------------+---------------------------------------------+
|2017-01-01 00:00:03.0|[2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0]|
|2017-01-01 00:00:06.0|[5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0]|
|2017-01-01 00:00:09.0|[8.0, 8.0, 8.0, 9.0, 9.0, 9.0]               |
+---------------------+---------------------------------------------+

Spark 1.6

We cannot use array as an argument inside collect_list() , so we'll just wrap the collect_list() calls inside array , instead of the other way around. We'll also slightly modify our udf because we won't be explicitly needing the rank column using this approach.

unpack_udf = udf(
    lambda l: [item for sublist in l for item in sublist]
)

df.select('*', rank().over(window).alias('rank')) \
  .groupBy("interval") \
  .agg(array(collect_list("a"),
             collect_list("b"),
             collect_list("c")).alias("vals")) \
  .withColumn("vals", unpack_udf("vals")) \
  .sort("interval") \
  .show(truncate = False)
+---------------------+---------------------------------------------+
|interval             |vals                                         |
+---------------------+---------------------------------------------+
|2017-01-01 00:00:03.0|[2.0, 3.0, 4.0, 2.0, 3.0, 4.0, 2.0, 3.0, 4.0]|
|2017-01-01 00:00:06.0|[5.0, 6.0, 7.0, 5.0, 6.0, 7.0, 5.0, 6.0, 7.0]|
|2017-01-01 00:00:09.0|[8.0, 9.0, 8.0, 9.0, 8.0, 9.0]               |
+---------------------+---------------------------------------------+

Note that vals column is now ordered in a different way, yet consistently thanks to the window function we defined earlier.