kafka use-case for error topics

Question

I'm trying to put a pipeline in place and I just realized I don't really know why there will be an error and why there will be an error topic. There is some metadata that I will be counting on to be certain values but other than that, is there anything that is a "typical" kafka error? I'm not sure what the "typcial" kafka error topic is used for. This is specifically for a streams application. Thanks for any help.

Answer 1

One example of an error topic in a streaming environment would be that it contains messages that failed to abide by their contract.. example: if your incoming events are meant to be in a certain json format, your spark application will first try to parse the event into a class that fits the events json contract.

If it is in the right format, it is parsed and the app continues.

If it is in the incorrect format, the parsing fails and the json string is sent to the error topic.

Other use cases could be to to send the event back to an error topic to be processed at a later time.. ie network issues connecting to other services.