Apache Flink：为 DataStream 添加侧输入 API

Question

In my Java application, I have three DataStreams . For example, for One stream data is consumed from Kafka, for another stream data is consumed from Apache Nifi. For these two streams Object type is different. For example, Stream-1 object type is Person, Stream-2 object type is Address.

The third one is the broadcast stream (for this data is consumed from Kafka).

Now I want to combine Stream-1 and Stream-2 in a Job class and want to split in the task process element. How to implement this?

Note: Stream-1 is mainstream and Stream-2 is side input. MainStream is continuously fetching data from Kafka. For Side Input, initially while the application is UP all table data is loaded from DB and then read new data when the table data is updated (not frequently).

Sample structure:

DataStream<Person> stream-1 = env.addSource(read data from kafka)....
DataStream<Address> stream-2 = env.addSource(read data from nifi)....
BroadcastStream<String> BroadCastStream = stream-3.broadcast(read data from kafka);

I was referred to as the following links.

FLIP-17 Side Inputs for DataStream API

jira/browse/FLINK-6131

My Use case is:

Join stream with slowly evolving data: The side input that we use for enriching is evolving over time (Data is read from DB). This can be done by waiting for some initial data to be available before processing the main input and the continuously ingesting new data into the internal side input structure as it arrives.

Answer 1

Based on the latest response, the recommendation by @Arvid was in fact what was needed here.

Core of the answer:

You can easily join stream1 and stream2 even if they have different types. Then you can add the broadcast to the result

Links to doc and example , and a relevant snippet from the doc (the example is too long to be included in here):

import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
 
...

DataStream<Integer> orangeStream = ...
DataStream<Integer> greenStream = ...

orangeStream.join(greenStream)
    .where(<KeySelector>)
    .equalTo(<KeySelector>)
    .window(TumblingEventTimeWindows.of(Time.milliseconds(2)))
    .apply (new JoinFunction<Integer, Integer, String> (){
        @Override
        public String join(Integer first, Integer second) {
            return first + "," + second;
        }
    });