如何创建一个独立于 Django 服务器的 kafka 消费者

Question

I am fairly new to Python. I have a django based web application where I am performing some tasks using Kafka. I am using confluent-kafka wrapper to communicate with the Kafka Broker.

So once a topic is created on the kafka broker , I assign a consumer to subscribe to this process, to make this non-blocking, I am using multiprocessing module so that the consumer runs on a separate thread and not block the main application thread.

However, I am not sure, what will happen if my server shuts down? Will the process still run like how the cronjobs still run even if the server is not running?

If not can someone please suggest me how should I trigger my consumer so that it is independent of my django server?

Answer 1

If you need your web server to actually get the data from Kafka, they need to be running together.

You probably don't want the consumer actively running in the background (as a separate process, not a thread) because then you're effectively not processing messages.

Depending on what your data is, you could use Kafka Connect on the same machine as Django to write the Kafka events to a database, then Django would query the database instead

Answer 2

bind the signals: signal.signal(signal.SIGTERM, signal_handler) signal.signal(signal.SIGINT, signal_handler) and use try-except blok and you can do a clean up afterwards.

to make consumer more independend, you can use django-celery-beat