Logstash plugin to output to Kafka not working

Question

I'm trying to send messages from Logstash to Kafka and keep getting the following error:

tail -f /var/log/logstash/logstash-plain.log
[2018-04-18T17:55:33,836][WARN ][org.apache.kafka.clients.NetworkClient] 
[Producer clientId=producer-1] Connection to node -1 could not be established. Broker may not be available.
[2018-04-18T17:55:46,095][INFO ][logstash.outputs.kafka   ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.1}
[2018-04-18T17:55:51,979][INFO ][logstash.outputs.kafka   ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.1}
[2018-04-18T17:56:46,196][INFO ][logstash.outputs.kafka   ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.1}]

This was working previously, but not now and I'm not sure what changed since the machine was untouched.

I can communicate to Kafka through the console producer on the machine Kafka is running on, as well as the machine Logstash is running on, and from a remote machine out of the VPC, so I don't believe this is a network error. To be sure, I put Logstash on the same machine as Kafka, but I get the same error.

I double checked and made sure that I'm at the latest version of Kafka, Logstash, and the logstash-output-kafka plugin, so I don't think it's a compatibility issue either. I have tried restarting Logstash and the broker to no avail.

Basically, I'm at a loss at how to solve this issue. Any help would be appreciated. Thanks.

Answer 1

Alright, I figured it out. It was a configuration issue in the logstash.conf file: When I changed the kafka output to the machine's local IP address, it works. It doesn't work with the public or private hostname, or the public IP address, it seems. Since I am using a conditional with multiple kafka topics like so:

output {
  kafka {
    bootstrap_servers => "10.0.0.XX:9092"
    codec => json
    topic_id => "test_topic"
  }
  if [type] == "ping" {
    kafka {
      bootstrap_servers => "10.0.0.XX:9092"
      codec => json
      topic_id => "logstash-ping"
    }
  }
  else if [type] == "snmp" {
    kafka {
      bootstrap_servers => "10.0.0.XX:9092"
      codec => json
      topic_id => "logstash-snmp"
    }
  }
}

it won't send to any of them if any one of them is not set right, despite connecting correctly from what I could see in the logstash logs.

For reference, my settings for listeners and advertised.listeners are like so:

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://ec2-XX-XX-XX-XX.compute-1.amazonaws.com:9092