ClickHouse Kafka Engine可以使用压缩的JSON消息吗？

Question

Can ClickHouse Kafka Engine consume compressed JSON messages from a Kafka topic, using JSONEachRow format?

Will it automatically decompress messages and then apply the JSON parsing? Does it need any special configuration for it?

EDIT My question is directed to whether compressing topics via set compression configuration on the Kafka topic would make ClickHouse not able to read these messages.

Answer 1

Kafka Engine consumes messages as they are - no more, no less. It's almost the same as reading from native files, with the exception that message boundaries might need special treatments. ClickHouse doesn't support reading compressed data files other than native serialized blocks. So the answer is no, it cannot consume compressed JSON messages.

Answer 2

Yes, as long as the ClickHouse client is able to understand Kafka topic compression. For example, earlier versions won't be able to understand ZStandard compression, but they would understand eg snappy. Whether it's JSON or something else, that's up to the parsing method specified. If these messages are themselves compressed (redundant though) then they will not be parsed correctly, as @Amos says.