[英]Quarkus Kafka Streams/Reactive Messaging Deserializing Exception
Hey so I was experimenting with both Kafka Streams and MP Reactive Messaging to read from a Kafka Topic and then to produce back to it.嘿,所以我正在尝试使用 Kafka Streams 和 MP Reactive Messaging 来读取 Kafka 主题,然后再返回给它。
The Kafka Streams error -卡夫卡流错误 -
org.apache.kafka.streams.errors.StreamsException: Deserialization exception handler is set to fail upon a deserialization error. If you would rather have the streaming pipeline continue after a deserialization error, please set the default.deserialization.exception.handler appropriately.
The Reactive Messaging error is similar, but basically the POJO that the messaging is getting deserialized to looks like this - Reactive Messaging 错误类似,但基本上消息被反序列化为的 POJO 看起来像这样 -
public class FinancialMessage {
public String user_id;
public String stock_symbol;
public String exchange_id;
public String trade_type;
public String date_created;
public String date_submitted;
public int quantity;
public double stock_price;
public double total_cost;
public int institution_id;
public int country_id;
public boolean compliance_services;
public boolean technical_validation;
public boolean schema_validation;
public boolean business_validation;
public boolean trade_enrichment;
}
Note that there is a default empty constructor and a constructor with all the fields.请注意,有一个默认的空构造函数和一个包含所有字段的构造函数。
import java.time.Instant;
import javax.enterprise.context.ApplicationScoped;
import javax.enterprise.inject.Produces;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.Consumed;
import org.apache.kafka.streams.kstream.GlobalKTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.kstream.Produced;
import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
import org.apache.kafka.streams.state.Stores;
import io.quarkus.kafka.client.serialization.JsonbSerde;
import io.quarkus.kafka.client.serialization.JsonbSerializer;
@ApplicationScoped
public class ComplianceTopology {
private static final String KTABLE_TOPIC = "kstreams-ktable-topic";
private static final String INCOMING_TOPIC = "kstreams-incoming-test";
private static final String OUTGOING_TOPIC = "kstreams-outgoing-test";
@Produces
public Topology buildTopology() {
StreamsBuilder builder = new StreamsBuilder();
JsonbSerde<FinancialMessage> financialMessageSerde = new JsonbSerde<>(FinancialMessage.class);
builder.stream(
INCOMING_TOPIC,
Consumed.with(Serdes.Integer(), financialMessageSerde)
)
.filter(
(key, message) -> checkCompliance(message)
)
.mapValues (
checkedMessage -> performComplianceCheck(checkedMessage)
)
.to (
INCOMING_TOPIC,
Produced.with(Serdes.Integer(), financialMessageSerde)
);
return builder.build();
}
public boolean checkCompliance (FinancialMessage rawMessage) {
return (rawMessage.compliance_services);
}
public FinancialMessage performComplianceCheck(FinancialMessage checkedMessage) {
checkedMessage.compliance_services = false;
return checkedMessage;
}
}
However, I guess it's called a "poison pill" but a single message produced from MQ with a payload of 'Aloha' breaks it and I can't deserialize it.但是,我猜它被称为“毒丸”,但是从 MQ 产生的一条带有“Aloha”有效负载的消息会破坏它,我无法反序列化它。 I'm guessing the reason for this is that 'Aloha' is not recognized as a String since it's in single quotes.
我猜这是因为 'Aloha' 不被识别为字符串,因为它是单引号。 I don't have access to how that data is sent since it was sent by ways of MQ.
我无权访问该数据的发送方式,因为它是通过 MQ 发送的。 Is there a way to skip processing this non-deserializable message and just continue with processing from the topic?
有没有办法跳过处理这个不可反序列化的消息并继续从主题处理?
As indicate in the error message如错误消息中所示
please set the default.deserialization.exception.handler appropriately
you can configure a different deserialization exception handler to skip over messages that cannot be deserialized.您可以配置不同的反序列化异常处理程序来跳过无法反序列化的消息。
Check out the docs for more details: https://kafka.apache.org/documentation/streams/developer-guide/config-streams.html#default-deserialization-exception-handler查看文档了解更多详细信息: https://kafka.apache.org/documentation/streams/developer-guide/config-streams.html#default-deserialization-exception-handler
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.