Starting Logstash, connecting it to kafka producer

Question

I have installed Elasticsearch and Logstash as a primary component.

1. Started the Kafka producer that creates records every 4 seconds
2. systemctl status logstash , says the logstash is running
3. From the bin folder of logstash, when I run the below command to listen to a topic, it gives an error bash: “localhost:9092”: Permission denied

/usr/share/logstash/bin$ logstash -e ‘input { kafka { bootstrap_servers => “localhost:9092” topics => “Customer_demo” } } filter { json { source => “message”} } output { elasticsearch { hosts => [“localhost:9200”] index => “customer-demo” } stdout {} }

Answer 1

THe producer is not running now, however, the plain logs are- I will take a snapshot by getting the producer up.

[2021-02-03T15:57:25,387][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][main][5f6aa7eb9c5d8e0614720d7b335f8771464d38e2be323dcdb318ca41dfd3e53a] [Consumer clientId=logstash-0, groupId=logstash] Group coordinator tej-Vostro-3590:9092 (id: 2147483647 rack: null) is unavailable or invalid, will attempt rediscovery
[2021-02-03T15:57:25,387][INFO ][org.apache.kafka.clients.FetchSessionHandler][main][5f6aa7eb9c5d8e0614720d7b335f8771464d38e2be323dcdb318ca41dfd3e53a] [Consumer clientId=logstash-0, groupId=logstash] Error sending fetch request (sessionId=786233676, epoch=9143) to node 0: {}.
org.apache.kafka.common.errors.DisconnectException: null
[2021-02-03T15:57:25,827][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][main][5f6aa7eb9c5d8e0614720d7b335f8771464d38e2be323dcdb318ca41dfd3e53a] [Consumer clientId=logstash-0, groupId=logstash] Discovered group coordinator tej-Vostro-3590:9092 (id: 2147483647 rack: null)
[2021-02-03T16:38:01,572][INFO ][org.apache.kafka.clients.FetchSessionHandler][main][5f6aa7eb9c5d8e0614720d7b335f8771464d38e2be323dcdb318ca41dfd3e53a] [Consumer clientId=logstash-0, groupId=logstash] Error sending fetch request (sessionId=1455350722, epoch=2665) to node 0: {}.
org.apache.kafka.common.errors.DisconnectException: null
[2021-02-03T16:38:01,585][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][main][5f6aa7eb9c5d8e0614720d7b335f8771464d38e2be323dcdb318ca41dfd3e53a] [Consumer clientId=logstash-0, groupId=logstash] Group coordinator tej-Vostro-3590:9092 (id: 2147483647 rack: null) is unavailable or invalid, will attempt rediscovery
[2021-02-03T16:38:03,556][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][main][5f6aa7eb9c5d8e0614720d7b335f8771464d38e2be323dcdb318ca41dfd3e53a] [Consumer clientId=logstash-0, groupId=logstash] Discovered group coordinator tej-Vostro-3590:9092 (id: 2147483647 rack: null)
[2021-02-03T16:38:03,560][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][main][5f6aa7eb9c5d8e0614720d7b335f8771464d38e2be323dcdb318ca41dfd3e53a] [Consumer clientId=logstash-0, groupId=logstash] Group coordinator tej-Vostro-3590:9092 (id: 2147483647 rack: null) is unavailable or invalid, will attempt rediscovery
[2021-02-03T16:38:03,664][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][main][5f6aa7eb9c5d8e0614720d7b335f8771464d38e2be323dcdb318ca41dfd3e53a] [Consumer clientId=logstash-0, groupId=logstash] Discovered group coordinator tej-Vostro-3590:9092 (id: 2147483647 rack: null)