Python Kafka - max messages in file

Question

Is there any way to get just one message, save it to a file, and terminate the script? I'd like to loop this over so that it saves me each message to a separate file. From the cmd level, I had a parameter:

max-messages 1 

but in python I haven't found anything like this. Code below:

from kafka import KafkaConsumer
import sys
import datetime

now_day = datetime.datetime.now().strftime("%Y-%m-%d")
print(now_day)

directory = "D:/Kafka/test/files_" + now_day + "/earnix_topic_"
print(directory)

now_datetime = datetime.datetime.now().strftime("%Y-%m-%d_%H%M")
print("Current date: " + now_datetime)

full_path = directory + now_datetime + ".txt"
print(full_path)

bootstrap_servers = ['prod-kafka-wrk01:0000','prod-kafka-wrk02:0001','prod-kafka-wrk03:0003']

# Define topic name from where the message will recieve
topicName = 'notify.products.client.topic'

consumer = KafkaConsumer(topicName,  group_id ='groupABasdsadC',bootstrap_servers = bootstrap_servers)

# Read and print message from consumer
for msg in consumer:
   file = open(full_path, "w+")
   file.write(str(msg.value))
   file.close()

sys.exit()

Answer 1

You implicitly use the __iter__() or __getitem__() method on the consumer if you use:

for msg in consumer:
  #do something...

You can use the built-in next() function that will also make use of the __iter__() method:

msg = next(consumer)

For more on python iterable types, see: https://www.pythonlikeyoumeanit.com/Module2_EssentialsOfPython/Iterables.html

Or you can ignore the iterator adapter and use the poll(...) method, that will return a smaller list of messages that are currently in the topic. See https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html

Answer 2

You need to indent sys.exit() into the loop if you want to consume only one record and end the script