卡夫卡在重新平衡转移的分区时是否应用压缩

Question

Our new kafka cluster was running out of space faster than anticipated, so I took some measures to mitigate:

1) I turned on snappy compression for the topics in question: this was done on the broker-side with a "kafka-topics --config compression.type=snappy" command, so as not to interrupt producers

2) I doubled the number of kafka nodes in the cluster

3) I rebalanced the cluster.

These steps worked and we have bought some time. I know that [1] only applies to newly-produced data, but I was wondering if partitions which were rebalanced to the new nodes would be compressed when they arrived at their new broker, or whether they would retain their original form (there was no producer compression, so they were all uncompressed). The old nodes had some other stuff on them which makes it difficult to infer from a simple disk usage analysis.

Answer 1

在重新分配分区期间，记录将被压缩并写入新代理。