使用 Flink 同步处理 2 个流

Question

I have 2 streams A and B.

I started consuming both A and B.

Stream A gets a record only at 59th second of every minute.

Stream B get a record any second of the minute.

I want to process such that both the streams are in sync.

Example: From Stream A after 10:01:59 I will receive a record at 10:02:59, until 10:02:59 I don't want to read anything from Stream B as well.

Can this be achieved in Flink?

Answer 1

You can't not read records from a stream in Flink, but you can drop (or save in state) records from a stream. So you could connect the two streams, and process with a CoFlatMap. When you get a record from Stream A, save it in state. When you get a record from Stream B, decide what to do with it, based on the state of Stream A.

Answer 2

Flink uses a push based model (this should change soon as sources and sinks are being refactored to a pull based model) to process elements downstream. This means that you cannot "wait until an event has arrived to pull in more data", and you must buffer that in some operator state in the meanwhile. Flink offers various state backends for you to utilize.

To give some visualization on kkruglers answer , given two streams, this is how we'd logically connect them and then use a ListState for one of them to be retrieved when another element arrives:

import org.apache.flink.api.common.state.{ListState, ListStateDescriptor}
import org.apache.flink.configuration.Configuration
import org.apache.flink.streaming.api.functions.co.CoProcessFunction
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala._
import org.apache.flink.util.Collector

import scala.collection.JavaConverters._ 

object Test {
  def main(args: Array[String]): Unit = {
    val env = StreamExecutionEnvironment.createLocalEnvironment()
    val streamA = env.fromCollection(List(1, 2, 3))
    val streamB = env.fromCollection(List("a", "b", "c"))

    streamA
      .connect(streamB)
      .process {
        new CoProcessFunction[Int, String, (Int, String)] {
          var myStateA: ListState[Int] = _

          override def open(parameters: Configuration): Unit = {
            myStateA = getRuntimeContext.getListState[Int](
              new ListStateDescriptor[Int]("my_state", classOf[Int])
            )
          }

          override def processElement1(
              value: Int,
              ctx: CoProcessFunction[Int, String, (Int, String)]#Context,
              out: Collector[(Int, String)]
          ): Unit = {
            myStateA.add(value)
          }

          override def processElement2(
              value: String,
              ctx: CoProcessFunction[Int, String, (Int, String)]#Context,
              out: Collector[(Int, String)]
          ): Unit = {
            val list = myStateA.get().iterator().asScala.toList
            val intFromState = list.headOption
            intFromState match {
              case Some(myInt) =>
                out.collect((myInt, value))
              case None => ()
            }

            myStateA.update(list.tail.asJava)
          }
        }
      }
  }
}

Note this implementation is simplified. There is no guarantee here for the order of arrival of elements, which you'd need to add to your state and implementation. You could additionally use Timers , and thus register a timer per event that enters the stream, that as an indication for when new data has arrived.