KStream-KStream內連接拋出java.lang.ClassCastException

Question

在過程方法@StreamListener ，我校映射到KStream人KStream並通過。通過（）方法來填充主題的“人”，從我產生KStream內的另一種方法過程1 @StreamListener 。

MianApplication.java

@SpringBootApplication
public class KafkaStreamsTableJoin {

    public static void main(String[] args) {
        SpringApplication.run(KafkaStreamsTableJoin.class, args);
    }

    @EnableBinding(KStreamProcessorX.class)
    public static class KStreamToTableJoinApplication {

        @StreamListener
        public void process(@Input("school") KStream<SchoolKey, School> schools) {  

            schools.map((schoolKey, school) -> {
                return KeyValue.pair(new PersonKey("Adam", "Smith", schoolKey.getId()), new Person(12));
            })
            .through("person", Produced.with(new PersonKeySerde(), new PersonSerde()));
        }

        @StreamListener
        public void process1(@Input("school_1") KStream<SchoolKey, School> schools, @Input("person") KStream<PersonKey, Person> persons) {

            schools.selectKey((schoolKey, school) -> schoolKey.getId())
                    .join(persons.selectKey((personKey, person) -> personKey.getId()),
                            (school, person) -> {
                                System.out.println("school_app2= " + school + ", person_app2= " + person);
                                return null;
                            },
                            JoinWindows.of(Duration.ofSeconds(1)),
                            Joined.with(Serdes.Integer(), new SchoolSerde(), new PersonSerde())
                    );
        }
    }

    interface KStreamProcessorX {

        @Input("person")
        KStream<?, ?> inputPersonKStream();

        @Input("school")
        KStream<?, ?> inputSchoolKStream();

        @Input("school_1")
        KStream<?, ?> inputSchool1KStream();

    }
}

在方法process1中，這個KStream需要與另一個KStream連接，但是我得到以下異常：

Exception in thread "stream-join-sample_2-654e8060-5b29-4694-9188-032a9779529c-StreamThread-1" java.lang.ClassCastException: class kafka.streams.join.School cannot be cast to class kafka.streams.join.Person (kafka.streams.join.School and kafka.streams.join.Person are in unnamed module of loader 'app')
    at org.apache.kafka.streams.kstream.internals.AbstractStream.lambda$reverseJoiner$0(AbstractStream.java:98)
    at org.apache.kafka.streams.kstream.internals.KStreamKStreamJoin$KStreamKStreamJoinProcessor.process(KStreamKStreamJoin.java:94)
    at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:117)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:183)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:162)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:122)
    at org.apache.kafka.streams.kstream.internals.KStreamJoinWindow$KStreamJoinWindowProcessor.process(KStreamJoinWindow.java:55)
    at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:117)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:183)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:162)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:122)
    at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:87)
    at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:366)
    at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:199)
    at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:420)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:889)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:804)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:773)

我認為這個例外與不正確的serde相關，但我無法弄清楚哪個serde正在創建問題以及如何修復它。 或者是在方法過程中的映射過程中，是否觸發了重新分區，這與錯誤的serde有關？

POJO和Serde：

Person.java

public class Person {

    private double age;

    public Person() {
    }

    public Person(double age) {
        this.age = age;
    }

    @JsonGetter("age")
    public double getAge() {
        return age;
    }

    @JsonSetter("age")
    public void setAge(double age) {
        this.age = age;
    }

    @Override
    public String toString() {
        return "Person{" +
                "age=" + age +
                '}';
    }
}

PersonSerde.java

public class PersonSerde extends Serdes.WrapperSerde<Person> {
    public PersonSerde () {
        super(new JsonSerializer<>(), new JsonDeserializer<>(Person.class));
    }
}

PersonKey.java

public class PersonKey {

    private String firstName;
    private String lastName;
    private int id;

    public PersonKey() {
    }

    public PersonKey(String firstName, String lastName, int id) {
        this.firstName = firstName;
        this.lastName = lastName;
        this.id = id;
    }

    @JsonGetter("firstName")
    public String getFirstName() {
        return firstName;
    }

    @JsonSetter("firstName")
    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }

    @JsonGetter("lastName")
    public String getLastName() {
        return lastName;
    }

    @JsonSetter("lastName")
    public void setLastName(String lastName) {
        this.lastName = lastName;
    }

    @JsonGetter("id")
    public int getId() {
        return id;
    }

    @JsonSetter("id")
    public void setId(int id) {
        this.id = id;
    }

    @Override
    public String toString() {
        return "PersonKey{" +
                "firstName='" + firstName + '\'' +
                ", lastName='" + lastName + '\'' +
                ", id=" + id +
                '}';
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;
        PersonKey personKey = (PersonKey) o;
        return id == personKey.id &&
                Objects.equals(firstName, personKey.firstName) &&
                Objects.equals(lastName, personKey.lastName);
    }

    @Override
    public int hashCode() {
        return Objects.hash(firstName, lastName, id);
    }
}

PersonKeySerde.java

public class PersonKeySerde extends Serdes.WrapperSerde<PersonKey> {
    public PersonKeySerde () {
        super(new JsonSerializer<>(), new JsonDeserializer<>(PersonKey.class));
    }
}

學校類的serde和pojo類似於Person類。

application.yml

spring.application.name: stream-join-sample

spring.cloud.stream.bindings.school:
  destination: school
  contentType: application/json
  consumer:
    useNativeDecoding: false
spring.cloud.stream.kafka.streams.bindings.school:
  consumer:
    keySerde: kafka.streams.serde.SchoolKeySerde
    valueSerde: kafka.streams.serde.SchoolSerde
    application-id: stream-join-sample_1

spring.cloud.stream.bindings.person:
  destination: person
  contentType: application/json
  consumer:
    useNativeDecoding: false
spring.cloud.stream.kafka.streams.bindings.person:
  consumer:
    keySerde: kafka.streams.serde.PersonKeySerde
    valueSerde: kafka.streams.serde.PersonSerde
    application-id: stream-join-sample_2

spring.cloud.stream.bindings.school_1:
  destination: school
  contentType: application/json
  consumer:
    useNativeDecoding: false
spring.cloud.stream.kafka.streams.bindings.school_1:
  consumer:
    keySerde: kafka.streams.serde.SchoolKeySerde
    valueSerde: kafka.streams.serde.SchoolSerde
    application-id: stream-join-sample_2

spring.cloud.stream.kafka.streams.binder:
  brokers: localhost
  configuration:
    default.key.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
    default.value.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
    commit.interval.ms: 100

具有可重復步驟的樣品應用 ：

Answer 1

我從GitHub下載了你的代碼來深入研究它，結果發現它實際上是用過的JsonSerializer / JsonDeserializer中的一個bug。 類型（ School ， Person ， PersonKey ， SchoolKey ）在記錄標題中編碼，但標題永遠不會被清除。 每次類型更改時，只會附加一個新標頭（標頭鍵不是唯一的，並且允許重復）。

對於某些記錄，相同的類型只是被編碼多次，因此，這部分代碼可以工作。 但是，對於某些情況，編碼不同類型，並且在從主題讀取數據時“隨機”選擇一種類型（找到的第一個標題）。 在加入之前發生，但是從重新分區主題接收數據時。 如果選擇了錯誤的類型，代碼將在稍后使用ClassCastException崩潰。

新答案：

在對此票證進行討論之后，您應該禁用該類型信息，通過以下方式將其寫入記錄標題中： https：//github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/685

props.put(JsonSerializer.ADD_TYPE_INFO_HEADERS, false);

請注意，所有Serdes手動創建，即通過調用new必須手動配置：

Map<String, Object> config = new HashMap<>();
config.put(JsonSerializer.ADD_TYPE_INFO_HEADERS, false);

PersonKeySerde personKeySerde = new PersonKeySerde();
personKeySerde.configure(config, true);

PersonSerde personSerde = new PersonSerde();
personSerde.configure(config, false);

// ...
.through("person", Produced.with(personKeySerde, personSerde));

原答案：

作為一種變通方法，可以更換map和selectKey()與transform()並清除頭中transform() 這是一個黑客。 您應該JsonSerializer SpringBoot項目提交票證，以便他們可以修復JsonSerializer / JsonDeserializer 。

以下代碼刪除標頭並確保使用正確的類型，避免ClassCastException ：

@SpringBootApplication
public class KafkaStreamJoinApplication {

    public static void main(String[] args) {
        SpringApplication.run(KafkaStreamJoinApplication.class, args);
    }

    @EnableBinding(KStreamProcessorX.class)
    public static class KafkaKStreamJoinApplication {

        @StreamListener
        public void process(@Input("school") KStream<SchoolKey, School> schools) {
            // replace map() with transform()
            schools.transform(new TransformerSupplier<SchoolKey, School, KeyValue<PersonKey, Person>>() {
                @Override
                public Transformer<SchoolKey, School, KeyValue<PersonKey, Person>> get() {
                    return new Transformer<SchoolKey, School, KeyValue<PersonKey, Person>>() {
                        ProcessorContext context;

                        @Override
                        public void init(final ProcessorContext context) {
                            this.context = context;
                        }

                        @Override
                        public KeyValue<PersonKey, Person> transform(final SchoolKey key, final School value) {
                            // clear all headers; would be sufficient to only remove type header
                            for (Header h : context.headers().toArray()) {
                                context.headers().remove(h.key());
                            }
                            // same a "old" map code:
                            return KeyValue.pair(new PersonKey("Adam", "Smith", key.getId()), new Person(12));
                        }

                        @Override
                        public void close() {}
                    };
                }})
                .through("person", Produced.with(new PersonKeySerde(), new PersonSerde()));
        }

        @StreamListener
        public void process1(@Input("school_1") KStream<SchoolKey, School> schools, @Input("person") KStream<PersonKey, Person> persons) {

            // replace selectKey() with transform()
            schools.transform(new TransformerSupplier<SchoolKey, School, KeyValue<Integer, School>>() {
                @Override
                public Transformer<SchoolKey, School, KeyValue<Integer, School>> get() {
                    return new Transformer<SchoolKey, School, KeyValue<Integer, School>>() {
                        ProcessorContext context;

                        @Override
                        public void init(final ProcessorContext context) {
                            this.context = context;
                        }

                        @Override
                        public KeyValue<Integer, School> transform(final SchoolKey key, final School value) {
                            // clear all headers; would be sufficient to only remove type header
                            for (Header h : context.headers().toArray()) {
                                context.headers().remove(h.key());
                            }
                            // effectively the same as "old" selectKey code:
                            return KeyValue.pair(key.getId(), value);
                        }

                        @Override
                        public void close() {}
                    };
                }})
                // replace selectKey() with transform()
                .join(persons.transform(new TransformerSupplier<PersonKey, Person, KeyValue<Integer, Person>>() {
                    @Override
                    public Transformer<PersonKey, Person, KeyValue<Integer, Person>> get() {
                        return new Transformer<PersonKey, Person, KeyValue<Integer, Person>>() {
                            ProcessorContext context;

                            @Override
                            public void init(final ProcessorContext context) {
                                this.context = context;
                            }

                            @Override
                            public KeyValue<Integer, Person> transform(final PersonKey key, final Person value) {
                                // clear all headers; would be sufficient to only remove type header
                                for (Header h : context.headers().toArray()) {
                                    context.headers().remove(h.key());
                                }
                                // effectively same as "old" selectKey code:
                                return KeyValue.pair(key.getId(), value);
                            }

                            @Override
                            public void close() {}
                        };
                    }}),
                    (school, person) -> {
                        System.out.println("school_app2= " + school + ", person_app2= " + person);
                        return null;
                    },
                    JoinWindows.of(Duration.ofSeconds(1)),
                    Joined.with(Serdes.Integer(), new SchoolSerde(), new PersonSerde())
                );
        }
    }

    interface KStreamProcessorX {
        @Input("person")
        KStream<?, ?> inputPersonKStream();

        @Input("school")
        KStream<?, ?> inputSchoolKStream();

        @Input("school_1")
        KStream<?, ?> inputSchool1KStream();
    }
}

Answer 2

可能是主題或底層的更改日志主題中有某些陳舊數據？ 您是否可以嘗試使用新主題和不同的應用程序ID來查看它是否可以解決您的問題？

以下是要使用的示例配置：

spring.cloud.stream.bindings.school:
  destination: school-abc
spring.cloud.stream.kafka.streams.bindings.school:
  consumer:
    keySerde: kafka.streams.serde.SchoolKeySerde
    valueSerde: kafka.streams.serde.SchoolSerde
    application-id: stream-join-sample_diff_id_1

spring.cloud.stream.bindings.person:
  destination: person-abc
spring.cloud.stream.kafka.streams.bindings.person:
  consumer:
    keySerde: kafka.streams.serde.PersonKeySerde
    valueSerde: kafka.streams.serde.PersonSerde
    application-id: stream-join-sample_diff_id_2

spring.cloud.stream.bindings.school_1:
  destination: school-abc
spring.cloud.stream.kafka.streams.bindings.school_1:
  consumer:
    keySerde: kafka.streams.serde.SchoolKeySerde
    valueSerde: kafka.streams.serde.SchoolSerde
    application-id: stream-join-sample_diff_id_2

spring.cloud.stream.kafka.streams.binder:
  brokers: localhost
  configuration:
    default.key.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
    default.value.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
    commit.interval.ms: 100

請注意，我更改了主題名稱，應用程序ID等。您可能希望更新填充主題的任何生產者。

另外，請注意您不需要指定內容類型，將useNativeDecoding設置為false等，因為這些是當前版本的kafka流綁定器中的默認值。