Kafka中主要队列和优先级队列的数据排序

Question

I need to process product id's in order and is planning to use kafka for this, now in case of data loss from kafka or my code i am having all these product-id's in may database so if record is not processed in a given time lets say 24 hours I need to republish them in a queue but in priority manner as kafka does not have priority concept for data in queue I can have another queue that can act as priority queue.

Problem I am facing is I need ordering of products in priority queue as well. So if I have distributed in partitions based on hash and my consumers again process messages in order ie, maintaining queue for each thread in consumer and based on hash of product-id I can main distribute data among these in memory queue. But in case of 2 queues, 1 as primary queue and 1 priority I ordering among these queues as well. So data from both queue should go to the same consumer so that I can maintain ordering in my code.

Please suggest if I am on the wrong track or how should I proceed.

Answer 1

It can be done if you need. You will not loose data in kafka if you have reasonable retention policy and replication factor.

---

Still how to do it:

1.Setup:

You can have two topics. Let us call them normal and priority. You have the same amount of partitions on both, let us have 4. You have the same partitioning strategy on both, let say product id mod 4.

2.Producer:

Now you have the event with your product of id 3. It is sent to the normal topic partition 3. You don't receive it for any reason. You republish the event about the same product on the priority topic now using the same logic so it goes to the same partition.

3.Consumer:

On the consumer side you have to manually assign to the specific partitions. Let say you have two consumers. You just assign the first one partitions 0 and 1 for both topics (and listen to both topics). The second is assigned the remaining both partitions 2 and 3.

I hope this answers you question. Good luck!