一段时间后，elasticsearch会停止使用Tyre对新文档建立索引

Question

I have my website running, with ElasticSearch, using Tire.

In the background, I have a rake task that is looping over thousands of objects and creating one record for each of them.

At first, the records are indexed, and every time I refresh my website, I can see the number of records found go up. But at some point, it just stops. It saves the records to mysql, but does not index them anymore. Note that the site and the search are still up and running.

Why is that? Is there some kind of lock or...?

---

edit: Here are errors I see in elasticsearch.log

[2013-03-16 08:49:34,344][WARN ][netty.channel.socket.nio.AbstractNioSelector] Failed to accept a connection. java.io.IOException: Too many open files

---

[Thakos] [maps][2] failed to read latest segment infos on flush java.io.FileNotFoundException: /usr/local/share/elasticsearch/data/elasticsearch/nodes/0/indices/maps/2/index/segments_3 (Too many open files)

So I guess that's need, I need to look at how to deal with this error.

Answer 1

I basically missed an important part of the "Installation guide": http://www.elasticsearch.org/guide/reference/setup/installation.html

I only had like 1024 open file descriptors allowed for the user running elasticsearch, and I needed around 32k (max set to 64k).

With these settings, everything's running perfectly.