Mirror Maker 2 不遵循compression.type from source

Question

我正在嘗試使用 compression.type = gzip 鏡像主題，但消息未壓縮就到達目標集群。 我正在使用 kafka 連接器來運行 MM2。

我嘗試過這些設置但沒有成功：

compression.type = gzip

producer.compression.type = gzip

target.compression.type = gzip

我正在使用以下命令檢查目標集群中的消息：

/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /path_to_log/00000000000000000000.log --print-data-log | grep compresscodec

有人在 MM 2 上遇到過這個問題嗎？ 或者任何提示？

Answer 1

我能夠打開 GZIP 壓縮到目標集群 (Cluster2) 的 MM2 生產者。 我不知道這是否是唯一正確的方法，但我使用這樣的配置：

#########################################################################
# specify any number of cluster aliases
clusters=Cluster1, Cluster2

# connection information for each cluster
# This is a comma separated host:port pairs for each cluster
# for e.g. "A_host1:9092, A_host2:9092, A_host3:9092"
Cluster1.bootstrap.servers=server1.mynetwork.net:9095
Cluster2.bootstrap.servers=server2.mynetwork.net:9093

# enable and configure individual replication flows
Cluster1->Cluster2.enabled=true

# regex which defines which topics gets replicated. For eg "foo-.*"
Cluster1->Cluster2.topics=.*

# Reverse
Cluster2->Cluster1.enabled=false
Cluster2->Cluster1.topics=.*

# Workers configs (Cluster2)
Cluster2.producer.compression.type=gzip
#########################################################################

在MM2日志中，我們可以看到這2個生產者開啟了GZIP壓縮選項：

[2021-01-27 14:52:19,006] INFO ProducerConfig values:
        acks = -1
        batch.size = 16384
        bootstrap.servers = [server2.mynetwork.net:9093]
        buffer.memory = 33554432
        client.dns.lookup = default
        client.id = **connector-producer-MirrorSourceConnector-0**
        compression.type = gzip
.
.
.
--
[2021-01-27 14:52:19,023] INFO ProducerConfig values:
        acks = -1
        batch.size = 16384
        bootstrap.servers = [server2.mynetwork.net:9093]
        buffer.memory = 33554432
        client.dns.lookup = default
        client.id = connector-producer-MirrorHeartbeatConnector-0
        compression.type = gzip
.
.
.

轉儲我的日志段我得到了這個：

baseOffset: 15190 lastOffset: 15190 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 1473759 CreateTime: 1611769946191 size: 110 magic: 2 compresscodec: GZIP crc: 1457322770 isvalid: true
| offset: 15190 CreateTime: 1611769946191 keysize: 20 valuesize: 10 sequence: -1 headerKeys: [] key:ClusterCluster1 payload: wD��O