如何将数据从风暴导出到弹性搜索？

Question

I stored my data into Apache Kafka. Then I able to use the data using Apache Storm spout and I processed the data. Now I want to export processed data into elastic search.

Answer 1

ElasticSearch provides Java client api available in maven repository.

If you already implemented pulling data from kafka to storm, all you have to do is implement one bolt that sends indexing request of that log to the elastic search.

Here I'm talking in traditional topology perspective.

For example, in your prepare method implementation, you create a transport client like this.

Client client = new TransportClient()
    .addTransportAddress(new InetSocketTransportAddress("host1", 9300))
    .addTransportAddress(new InetSocketTransportAddress("host2", 9300));

And in your execute method implementation, you send indexing request like this.

String json = "{" +
    "\"user\":\"kimchy\"," +
    "\"postDate\":\"2013-01-30\"," +
    "\"message\":\"trying out Elasticsearch\"" +
"}";

IndexResponse response = client.prepareIndex("twitter", "tweet")
    .setSource(json)
    .execute()
    .actionGet();

For more information, refer to http://www.elastic.co/guide/en/elasticsearch/client/java-api/current/index.html .

Answer 2

You can use ES Storm integration:

https://www.elastic.co/guide/en/elasticsearch/hadoop/current/storm.html

Or write back your data into Kafka and use LogStash to consume this queue and write to elasticsearch.