在Apache FLink中的不同窗口上计算指标

Question

I am using Apache Flink 1.2 and here's my question: I have a stream of data and I would like to compute a metric over a window of 1 day. Therefore I will write something like:

DataStream<Tuple6<Timestamp, String, Double, Double, Double, Integer>> myStream0 = 
            env.readTextFile("Myfile.csv")
            .map(new MyMapper())                // Parse the input
            .assignTimestampsAndWatermarks(new MyExtractor())   //Assign the timestamp of the event 
            .timeWindowAll(Time.days(1))    
            .apply(new average());  // compute average, max, sum

Now I would like to compute the same metrics over a window of 1 hour.

I can write same as before and specify Time.hours(1), but my concerns is that in this way apache flink reads two times the input file and does twice the work. I wonder if there is a way of doing all togheter (ie using the same stream).

Answer 1

You can compute hourly aggregates and from those the daily aggregates. This would look for a simple DataStream<Double> as follows:

DataStream<Double> vals = ... // source + timestamp extractor

DataStream<Tuple2<Double, Long>> valCnt = vals // (sum, cnt)
  .map(new CntAppender())     // Double -> Tuple2<Double, Long(1)>

DataStream<Tuple3<Double, Long, Long>> hourlySumCnt = valCnt // (sum, cnt, endTime)
  .timeWindowAll(Time.hours(1))
  // SumCounter ReduceFunction sums the Double and Long field (Long is Count)
  // WindowEndAppender WindowFunction adds the window end timestamp (3rd field)
  .reduce(new SumCounter(), new WindowEndAppender())   

DataStream<Tuple2<Double, Long>> hourlyAvg = hourlySumCnt // (avg, endTime)
  .map(new SumDivCnt()) // MapFunction divides Sum by Cnt for average

DataStream<Tuple3<Double, Long, Long>> dailySumCnt = hourlySumCnt // (sum, cnt, endTime)
  .map(new StripeOffTime()) // removes unnecessary time field -> Tuple2<Double, Long>
  .timeWindowAll(Time.days(1))
  .reduce(new SumCounter(), new WindowEndAppender()) // same as above

DataStream<Tuple2<Double, Long>> dailyAvg = dailySumCnt // (avg, endTime)
  .map(new SumDivCnt()) // same as above

So, you basically compute sum and count for each hour, and based on that result you

1. compute the hourly average
2. compute daily sum and count and the daily average

Note, that I am using a ReduceFunction instead of a WindowFunction for the sum and count computation, because a ReduceFunction is eagerly applied, ie, all records of the window are not collected but immediately aggregated. Hence the state that needs to be maintained is a single record.