How to register a TypeSerializer with Flink?

Question

I'm using Flink with Scala, and to get a better support for state migration I decided to move away from "let Flink handle everything with introspection" to having more explicit serialization/deserialization. This way when I make a change to my classes, I can decide how to handle the differences in my ser/deser.

So I decided to go with Avro, mostly following advices I found in this article: https://medium.com/wbaa/making-sense-of-apache-flink-state-migration-with-scala-and-avro-69091c232646

And taking inspiration from this example project: https://github.com/mrooding/flink-avro-state-serialization

Reading the article I can understand how to use a custom serializer when I am creating a state explicitely within a function.

For example I could write this when defining a state:

  private[this] lazy val stateDescriptor: ValueStateDescriptor[Product] =
    new ValueStateDescriptor[Product]("product-join", Product.serializer)
  private[this] lazy val state: ValueState[Product] = getRuntimeContext.getState(stateDescriptor)

The problem is, I don't know how to register the serializer when Flink implicitely creates a state from my pipeline. For example, if I create a FlatMapFunction[T, O] Flink will create a state with types T and O to save the instances in-between blocks. And in this case, it will still do introspection because I don't know how to tell Flink to use my TypeSerializer instead.

I found documentation to register a Kryo type with the StreamExecutionEnvironment , but not a TypeSerializer . My goal is not to use Kryo but Avro directly.

So how would I go to register a TypeSerializer globally on my environment, so Flink knows to use it instead of using introspection?

Answer 1

You need to implement a TypeInfoFactory and annotate your type with @TypeInfo. The factory needs to return a TypeInformation, which acts as a sort-of factory for TypeSerializers.