Kafka多个制作人写同一主题 - 消息和数据突发的排序

Question

I am trying to learn about Kafka. Say I have multiple producer each writing to same Topic. (Cant have more topics because of design and we using avro for serialization) Because our message is too big, we need to divide it into small parts and we send it to kafka.

In this scenario - Can mesages from different Producers can intermix ? How can I avoid this scenario ? any ideas

Also , we have huge databurst, like there will be 1000s of message in 2 minutes, then very few message for next 5 to 7 minutes, what can we do in such scenarios

Answer 1

Because our message is too big, we need to divide it into small parts and we send it to kafka.

Do you? You runned basic tests and had some issues? Have you tried to adjust buffers? I'm pretty sure that kafka can handle relatively big messages (tens of megabytes) without much hassle. In fact, you will likely get a better throughput, comparing to the huge number of tiny messages.

Also , we have huge databurst, like there will be 1000s of message in 2 minutes, then very few message for next 5 to 7 minutes, what can we do in such scenarios

Single kafka broker on a modern server machine can easily handle ~20k-40k messages/ per second (batched of 1000 messages, each 2k size, sync mode). I don't see a problem there.

Can messages from different Producers can intermix ?

Yes, they might intermix. In fact, this moment is not well defined by the kafka protocol and particular implementation may change this behaviour, so for stable solution take a look at the below section.

How can I avoid this scenario?

Kafka has a concept of partitions: each topic by default has 1 partition, each partition can be thought as a unit of parallelism. Setup appropriate partitioner, such that each producer writes to it's own partition, in isolated manner.