ELK logstash cant create index in ES

Question

after following this tuto ( https://www.bmc.com/blogs/elasticsearch-logs-beats-logstash/ ) in order to use logstash to analyze some log files, my index was created fine at the first time, then I wanted to re-index new files with new filters and new repositories so I deleted via "curl XDELETE" the index and now when I restart logstash and filebeat, the index is not created anymore.. I dont see any errors while launching the components. Do I need to delete something else in order to re-create my index?

Answer 1

Ok since my guess (see comments) was correct, here's the explanation:

To avoid that filebeat reads and publishes lines of a file over and over again, it uses a registry to store the current state of the harvester:

The registry file stores the state and location information that Filebeat uses to track where it was last reading.

As you stated, filebeat successfully harvested the files, sent the lines to logstash and logstash published the events to elasticsearch which created the desired index. Since filebeat updated its registry, no more lines had to be harvested and thus no events were published to logstash again, even when you deleted the index. When you inserted some new lines, filebeat reopened the harvester and published only the new lines (which came after the "registry checkpoint") to logstash.

The default location of the registry file is ${path.data}/registry (see Filebeat's Directory Layout Overview ).

... maybe the curl api call is not the best solution to restart the index

This has nothing to do with deleting the index. Deleting the index happens inside elasticsearch. Filebeat has no clue about your actions in elasticsearch.

Q: Is there a way to re-create an index based on old logs?

Yes, there are some ways you should take into consideration:

• You can use the reindex API which copies documents from one index to another. You can update the documents while reindexing them into the new index.
• In contrast to the reindex you can use the update by query API to update documents that will remain in the original index.
• Lastly you could of course delete the registry file. However this could cause data loss. But for development purposes I guess that's fine.

Hope I could help you.