Multiprocessing with python kafka consumers

Question

I am trying to make an application in python that has 1 topic (demo-topic) and 1 partition. In this topic messages are pushed randomly I have 1 consumer (consumer1) (demo-group) that uses this messages to makes some background calculations (that take some time).

Having this application on amazon i want to be able to scale it (when computation takes to long) in the way that the newly created machine will have another consumer (consumer 2) from the same group (demo-group) reading in the same topic (demo-topic) but in the way that they start splitting the load (consumer 1 takes some load and consumer 2 takes the rest but they never get the same messages)

After the surge of data comes to an halt, second machine is decommissioned and consumer 1 takes all the load again.

Is this even possible to do (without adding before hand more partitions) . Is there a workaround??

Thank you

Answer 1

You cannot have multiple consumers within the same group consume at the same time from the same partition. If you subscribe a second consume within the same group to the same partition, it will act as a hot standby and won't be consuming any messages until the first one stops.

The best solution is to add partitions to your topic. That way, you can add consumers when you see a surge in traffic and remove them when traffic slows down. Kafka will do all load balancing for you.

Answer 2

You could do this, but shouldn't.

The basic unit of parallelism in Kafka is the partition: in a consumer group, each consumer reads from one or more partitions and consumers do not share partitions. In order to share a partition, you would need to use a tool like ZooKeeper to lock access to the partition (and keep track of each consumer's position).

The use case that you're describing is better served by SQS and an Auto-scaling group.