如何让多个 Mirror Maker 节点分担复制负载？

Question

I am trying to enable replication between two different Kafka clusters (cluster A and cluster B). I do not require active/active replication. Going from A → B is enough for me.

To achieve that goal I plan to set up a Mirror Maker 2 cluster such as data flows as follows:

cluster A → Mirror Maker 2 cluster → cluster B

Cluster A has close to 3k partitions and it is part of a business-critical application so I have to guarantee speedy replication between A and B.

I started experimenting with setting up a dedicated mirror maker cluster (ie using connect-mirror-maker.sh ).

KIP-382 mentions that “In this mode, MirrorMaker does not require an existing Connect cluster. Instead, a high-level driver manages a collection of Connect workers.”. However, I am struggling with how to make multiple nodes communicate amongst themselves to divide up the load.

I have tried multiple configs such as listeners , rest.advertised.host.name , rest.advertised.port , and others.

None of them seem to enable communication between MM2 workers. Can someone point me to an example configuration on how to achieve that?

Answer 1

After digging a little into this I've found out that as of now (June/2022) there's no support for distributed mode in for the dedicated Mirror Maker cluster. If one wants to run Mirror Maker with multiple agents/workers, a connect cluster is needed. There's a KIP under consideration to change this behaviour.