[英]Kafka compression, how to limit it to some listed topics? How to use compressed.topics property using clients produce API?
I've enabled kafka message compression from client's producer API like : 我已经从客户端的生产者API启用了kafka消息压缩功能,例如:
configProperties.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip") configProperties.put(ProducerConfig.COMPRESSION_TYPE_CONFIG,“ gzip”)
It works well, however if we want to limit the above compression to some listed topics, we've notice a way : 效果很好,但是,如果我们想将上述压缩限制为某些列出的主题,我们会注意到一种方法:
configProperties.put("compressed.topics", "topicA,topicB") configProperties.put(“ compressed.topics”,“ topicA,topicB”)
from this link : https://cwiki.apache.org/confluence/display/KAFKA/Compression . 通过此链接: https : //cwiki.apache.org/confluence/display/KAFKA/Compression 。 But it is not working, means in my test case I've enabled compression and I didn't listed my topic in compressed.topics's args.
但这不起作用,这意味着在我的测试用例中,我启用了压缩功能,但没有在压缩的主题列表中列出主题。 still the compression been applied.
仍然应用了压缩。 please help.
请帮忙。
Compression is handled by several parameters in Kafka: 压缩由Kafka中的几个参数处理:
compression.type
configures the default for the entire cluster. compression.type
为整个群集配置默认值。 It allows the values gzip
, snappy
, lz4
, uncompressed
or producer
. gzip
, snappy
, lz4
, uncompressed
或producer
。 The first three are self-explanatory, they're just different compression methods (snappy/lz4 being the recommended ones), uncompressed
is self-explanatory, and the default producer
, which basically is just using whatever compression (or not) the producer decided to use. uncompressed
是不言自明的,默认的producer
,基本上是使用生产者决定的任何压缩(或不使用)使用。 In this case, Kafka just sees the messages as byte arrays, and doesn't try to decode them. gzip
, snappy
, lz4
, uncompresssed
or producer
. gzip
, snappy
, lz4
, uncompresssed
或producer
。 They're exactly the same, but act as overrides specifically for the topic you set them to. compression.type
, and the possible values are gzip
, snappy
, lz4
and none
, where this last one is the default. compression.type
,以及可能的值是gzip
, snappy
, lz4
和none
,在这最后一个是默认的。 You should be looking at the recent documentation for Kakfa. 您应该查看有关Kakfa的最新文档。 You don't specify which Kafka version you're using, but assuming a recent one, I would look at these docs:
您没有指定要使用的Kafka版本,但假设是最新版本,我将查看以下文档:
https://kafka.apache.org/20/documentation.html https://kafka.apache.org/20/documentation.html
In those, there is no mention of compression.topics
as a configuration option. 在这些文件中,没有提到
compression.topics
作为配置选项。 So, like @cricket_007 mentioned, this is an outdated/stale wiki page that you shouldn't be using. 因此,就像提到的@ cricket_007一样,这是您不应该使用的过时/过时的Wiki页面。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.