kafka - 可以将python程序连接到hadoop集群外的Kafka吗？

Question

I would like to write a python program which reads a file and writes the data to Kafka as a producer. Kafka is setup in the Hadoop cluster and Python will be running outside Hadoop cluster.

is it possible to connect from outside? do i need to install Kafka libraries for python? how do I need to install?

Answer 1

Easiest way to learn more about Kafka is to use http://landoop.com/docs/lenses/developers

You will need to run 1 docker - to bring up everything locally and then developer a Python app, using the appropriate Kafka libs that connect to Kafka via the Kafka APIs and produce messages to it

Once you've built your app - you can then package it and run it against your your Hadoop's Kafka brokers

Answer 2

is it possible to connect from outside?

Depends on the firewall rules, but yes, it is possible

do i need to install Kafka libraries for python? how do I need to install?

Yes. There are several Python libraries for Kafka. pip install kafka-python , for example.