将 UDP 消息推送到 Kafka 主题

Question

Is it possible for kafka producer to listen on certain ports ( UDP/TCP ) and then send that data to specified topic.

Do i need to develop a separate daemon which listens on the port and then send the data to kafka topic ? ( There are already a few github project available). Just wanted to make sure that we already a way to do it so that it becomes more robust and easy to scale. BAsically, relaying UDP packets into Kafka

Regards Sunil

Answer 1

As mentioned in comments, yes it is possible - several solutions can be found by doing a google search - here is one project, that lets you do what you want to do - out of the box:

https://github.com/agaoglu/udp-kafka-bridge

Answer 2

I would preferred to use rsyslogd: /etc/rsyslog.d/kafka.conf

module(load="imudp")
module(load="omkafka")

template(name="kafka_msg" type="string" string="%msg%")

ruleset(name="kafka") {
  :msg, !contains, "timestamp" ~
  action (
    type="omkafka"
    topic="nginx-accesslog"
    broker=["192.168.1.1:9092", "192.168.1.2:9092", "192.168.1.3:9092"]
    template="kafka_msg"
    confParam=["compression.codec=gzip"]
    partitions.auto="on"
  )
}

input(type="imudp" port="10514" Ruleset="kafka")