Kafka producer produce data to topic from PORT

Question

I'm new to Kafka.

I have a Linux machine in which port number 2552 getting data stream from external server.

I want to use Kafka producer to listen to that port and get the stream of data to a topic.

Answer 1

You don't say if the traffic on port 2552 is TCP or UDP but in general you can easily write a program that listens on that port, parses the data received into discrete messages, and then publishes the data to a Kafka Topic as Kafka messages (with or without keys) using the Kafka Producer API.

In some cases there is existing open source code that might already do this for you so you do not need to write it from scratch. If the port 2552 protocol is a well known protocol like for example the TCP or UDP call-logging protocol registered in IANA ( see ftp://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.txt ) then there might even be an existing Kafka Connector or Proxy that supports it. Search on GitHub for kafka-connect-[protocol] or take a look at the curated Connector list at https://www.confluent.io/product/connectors/

There may even be a generic TCP or UDP connector that you can use as a reference to configure or build your own for the specific protocol you are trying to ingest.

Answer 2

This is a complete hack, but would work for a sandbox example:

nc -l 2552 | ./bin/kafka-console-producer --broker-list localhost:9092 --topic test_topic

It uses netcat to listen on the TCP port, and pipe anything received to a Kafka topic.

A quick Google also turned up this https://github.com/dhanuka84/kafka-connect-tcp which looks to do a similar thing but more robustly, using the Kafka Connect API.