简体繁体 English

卡夫卡用例的错误主题

[英]kafka use-case for error topics

原文 2018-06-14 12:07:18 3 1 apache-kafka/ kafka-consumer-api/ kafka-producer-api/ apache-kafka-streams

I'm trying to put a pipeline in place and I just realized I don't really know why there will be an error and why there will be an error topic. 我试图建立一个管道，但我才意识到我真的不知道为什么会有错误，为什么会有错误主题。 There is some metadata that I will be counting on to be certain values but other than that, is there anything that is a "typical" kafka error? 我将依靠某些元数据来确定某些值，但除此之外，是否存在“典型” kafka错误？ I'm not sure what the "typcial" kafka error topic is used for. 我不确定“典型” kafka错误主题的用途。 This is specifically for a streams application. 这是专门针对流应用程序的。 Thanks for any help. 谢谢你的帮助。

1 个解决方案

One example of an error topic in a streaming environment would be that it contains messages that failed to abide by their contract.. example: if your incoming events are meant to be in a certain json format, your spark application will first try to parse the event into a class that fits the events json contract. 流环境中错误主题的一个示例是，它包含未能遵守其约定的消息。例如：如果传入的事件旨在采用某种json格式，则您的spark应用程序将首先尝试解析将事件转换为适合事件json合同的类。

If it is in the right format, it is parsed and the app continues. 如果格式正确，将对其进行解析，然后继续运行该应用程序。

If it is in the incorrect format, the parsing fails and the json string is sent to the error topic. 如果格式不正确，则解析失败，并将json字符串发送到错误主题。

Other use cases could be to to send the event back to an error topic to be processed at a later time.. ie network issues connecting to other services. 其他用例可能是将事件发送回错误主题，以便稍后处理。例如，网络问题连接到其他服务。