使用 python 连接 kafka 和 elasticsearch

Question

I'm new about using Kafka and elasticsearch. I've been trying to use Elastic search but I've some problem. I've grow up a docker compose file with all the images needed for building the environment then using kafka I've product into a specific topic the data and then I need to take from Kafka 's consumer data into a pub/sub system for sending data for the ingestion into elasticsearch. I implement all this using python. I've seen that into the port and localhost as ip elasticsearch appear instead for kibana in the page appear the following sentence: kibana server is not ready yet

the consumer python is something similar to it from which I take data from a topic:

from kafka import KafkaConsumer
# Import sys module
import sys

# Import json module to serialize data
import json

# Initialize consumer variable and set property for JSON decode
consumer = KafkaConsumer ('JSONtopic',bootstrap_servers = ['localhost:9092'],
value_deserializer=lambda m: json.loads(m.decode('utf-8')))
for message in consumer:
 print("Consumer records:\n")
 print(message)
 print("\nReading from JSON data\n")
 print("Name:",message[6]['name'])
 print("Email:",message[6]['email'])
 # Terminate the script
 sys.exit()

The goal is to use elasticsearch for doing analysis so I need to use it as backend as for visualize data into kibana. It could be really appreciate also a tutorial to follow for understanding what I should do for link this informations. (Ps data follow without problem from a topic to another one but the problem is to take this information and insert into elastic and have the possibility to visualize these informations)

Answer 1

If you're pushing data from Kafka to Elasticsearch then doing it with the Consumer API is typically not a good idea, since there are tools that exist that do it much better and handle more functionality.

For example:

• Kafka Connect (eg https://rmoff.dev/kafka-elasticsearch-video )
• Logstash