如何使用 confluent-kafka-python 删除主题

Question

I am using Kafka to make multiple micro services communicate with each other. Services are written in Python, and I use Confluent library to handle Kafka. At some point, I know that some Topics are just 'over', so that I could clean them automatically.

Is there a way to delete "Topics" thanks to Confluent library ? I cannot find any documentation on this...

Thanks

Answer 1

You can use the confluent Admin Api's to delete a topic

Example

Takes an AdminClient instance and a list of topics

def example_delete_topics(a, topics):
    """ delete topics """

    # Call delete_topics to asynchronously delete topics, a future is returned.
    # By default this operation on the broker returns immediately while
    # topics are deleted in the background. But here we give it some time (30s)
    # to propagate in the cluster before returning.
    #
    # Returns a dict of <topic,future>.
    fs = a.delete_topics(topics, operation_timeout=30)

    # Wait for operation to finish.
    for topic, f in fs.items():
        try:
            f.result()  # The result itself is None
            print("Topic {} deleted".format(topic))
        except Exception as e:
            print("Failed to delete topic {}: {}".format(topic, e))

Answer 2

To answer your question literally : no, AFAIK you can't delete topics using the client library.

This is possible using the AdminClient API

To address some of your follow up questions, I would suggest there is no point deleting these topics. If you're running a sandbox/prototype, then maybe you end up with lots of topics you want to clear down from experimenting. But once you move these microservices into a Production environment, each topic will serve a purpose, and perhaps it's finished processing the current set of data, but what about the next ? Unless you're doing something funky with your topic naming that ties it to some attributes in the data, then topics will be re-used.

Answer 3

I use this simple Python function:

def delete_kafka_topic(topic_name):
    call(["/usr/bin/kafka-topics", "--zookeeper", "zookeeper-1:2181", "--delete", "--topic", topic_name])

My team uses this in automated tests, where we want to be able to rerun tests, verify results, and not see results from previous test attempts.

Answer 4

Simplest example:

from kafka import KafkaAdminClient


admin_client = KafkaAdminClient(bootstrap_servers=['localhost:port'])
admin_client.delete_topics(topics=['test1', 'test2'])

Doc: https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#confluent_kafka.admin.AdminClient.delete_topics