无法连接到 MSK 上的 Kafka

Question

I have created a Kafka cluster on MSK and now I'm trying to connect to the cluster with python.

I wrote this shortcode:

from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers=['host1:9092', 'host2:9092'],
    value_serializer=lambda x: json.dumps(x).encode('utf-8'),
    api_version=(2, 4, 1)
)

producer.send('test', value={'hello':'world'})

The problem is that every time I running it I'm getting this error:

KafkaTimeoutError: Failed to update metadata after 60.0 secs.

I thought it might be related to Kafka creating topics so I added this line to the configuration.

auto.create.topics.enable=true

But I'm still getting the same error.

This is my full configuration file:

default.replication.factor=3
min.insync.replicas=2
num.io.threads=8
num.network.threads=5
num.partitions=1
num.replica.fetchers=2
socket.request.max.bytes=104857600
unclean.leader.election.enable=true
auto.create.topics.enable=true
zookeeper.connection.timeout.ms=5000

What am I missing here? I read somewhere that is may relate to SSL authentication but at any step, there wasn't any.pem file, .ca file, or anything like that.

Answer 1

You probably fail to connect to the MSK. The error you experience is a Timeout error.

One wrong assumption when first starting to use MSK is, you can connect to it from outside AWS network. That is a wrong assumption. AWS has a detailed document on how can you access your MSK cluster .

If you're not running your client from an EC2 instance inside your MSK VPC , you won't be able to reach it. Even if you open the relevant security rules in the MSK security group policy.

I have spend time trying multiple proxies to access the MSK from outside AWS, with no success. Just follow the guide I referred above, and it will probably fix your connection issue.

Also, If you're new to MSK I highly suggest you to go through the Getting Started tutorial, or at least Steps 5 ( Create a Topic ) and 6 ( Produce and Consume Data ).

Answer 2

kafka-python + SASL auth MSK worked for me

producer = KafkaProducer(
    bootstrap_servers=SASL_SERVERS,
    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
    api_version=(2,8,1),
    security_protocol='SASL_SSL',
    sasl_mechanism='SCRAM-SHA-512',
    sasl_plain_username=os.getenv('KAFKA_USER'),
    sasl_plain_password=os.getenv('KAFKA_PASS'),
    retries=1
)