Filter Flink tuples

Question

I'm writing a program for stream processing in Scala using Flink. I have a datastream which I first map to tuples containg json4s JValues. Now I want to filter these tuples based on these JValues. I thought this would be simple but I can't find any good example of how to filter Flink tuples by their columns. Does anyone know how to do this? Thanks

Answer 1

Instead of mapping to tuples, you could simply map to case classes and filter out unneeded stuff:

// StreamingJob.scala

...

val filteredEvents = content
      .map(x => Event.toCaseClass(x))
      .filter(x => x.value == true)

...

// Event.scala

case class Event(
                  id: String,
                  value: Int,
                )
object Event {
  implicit val formats = DefaultFormats

  def toCaseClass(str: String) =
    parse(str).extract[Event]
}

Answer 2

The question seems a little too undefined for me but maybe, does this not work?

// stream contains stuff like these in a flink tuple 
//(custom deserializer of array to tuple2???)
val jsonExample = """["foo", "bar"]"""

val stream: DataStream[Tuple2[JString, JString]] = ???
val filteredStream = stream.filter(x => x.getField(0).extract[String] == "foo")

Id say it would be better not to use flink tuples if you are writing scala though. Go for case classes or at least scala tuples maybe?