connect to a heroku kafka instance with kafka-python from outside heroku

Question

I have set up a heroku kafka instance, and I am trying to connect to it using the python consumer. I have the heroku environment in a file called .env by going heroku config -s > .env , and then load and export it before running this python program:

import os

from kafka import KafkaConsumer

for variable in ['KAFKA_TRUSTED_CERT', 'KAFKA_CLIENT_CERT', 'KAFKA_CLIENT_CERT_KEY']:
    with open(f'{variable}.txt', "w") as text_file:
        print(os.environ[variable], file=text_file)

consumer = KafkaConsumer('test-topic',
                         bootstrap_servers=os.environ['KAFKA_URL'],
                         security_protocol="SSL",
                         ssl_certfile="KAFKA_CLIENT_CERT.txt",
                         ssl_keyfile="KAFKA_CLIENT_CERT_KEY.txt"
)
for msg in consumer:
    print (msg)

I couldn't find any options that looked like they could load the certificates from a variable, so I put them all in files when I start the program.

When I run the program, it creates the temp files and doesn't complain, but doesn't print any messages.

When I write to the topic using the heroku cli like this

heroku kafka:topics:write test-topic "this is a test"

the python client doesn't print the message, but I can see the message by going

heroku kafka:topics:tail test-topic

Does anybody know what I am missing in the python consumer configuration?

Answer 1

In the official Heroku Kafka documentation:

https://devcenter.heroku.com/articles/kafka-on-heroku#using-kafka-in-python-applications

it states that using the Kafka helper is beneficial. If you look at the source code:

https://github.com/heroku/kafka-helper/blob/master/kafka_helper.py

one can see that they are writing the Kafka variables to files and creating an ssl_context.