将数据转储到elasticsearch时“打开的文件太多”

Question

我正在使用OSX Mavericks上的Elasticsearch-Exporter对elasticsearch执行转储：

node /usr/bin/node_modules/elasticsearch-exporter/exporter.js -j ${esIndexName} -f esbackup

我有一个运行两个节点的应用程序，它与应用程序节点一起总共增加了三个节点。 elasticsearch命令创建的节点是主节点。 当我对我的索引运行export命令时，我会在成功加载几秒后得到它：

2014-05-07T14:31:38.325-0700 [elasticsearch[Rancor][[es][1]: Lucene Merge Thread #0]] [WARN] merge.scheduler [][] - [Rancor] [es][1] failed to merge
 815 java.io.FileNotFoundException: /private/var/data/core/elasticsearch_me/nodes/0/indices/es/1/index/_f_es090_0.tip (Too many open files)

我尝试过以下方法：

launchctl limit 10000

sudo launchctl limit 40000 65000

elasticsearch soft nofile 32000

elasticsearch hard nofile 32000

将-XX:-MaxFDLimit添加到我的应用程序的jvm参数中

这些都没有解决我的问题。 偶尔负载将完成没有错误，但大多数时候我遇到错误。 有没有人对我的问题有什么想法/提示？

编辑：

$ launchctl limit cpu unlimited unlimited
filesize unlimited unlimited
data unlimited unlimited
stack 8388608 67104768
core 0 unlimited
rss unlimited unlimited
memlock unlimited unlimited
maxproc 709 1064
maxfiles 10000 10240

$ sudo launchctl limit Password: cpu unlimited unlimited
filesize unlimited unlimited
data unlimited unlimited
stack 8388608 67104768
core 0 unlimited
rss unlimited unlimited
memlock unlimited unlimited
maxproc 709 1064
maxfiles 40000 65000

Answer 1

好的 - 如果您在一台Mac上运行多个elasticsearch节点和node.js应用程序，我肯定会确保您的打开文件数量达到ES推荐的限制：

file descriptor

Make sure to increase the number of open files descriptors on the machine (or for the user running elasticsearch). Setting it to 32k or even 64k is recommended.

In order to test how many open files the process can open, start it with -Des.max-open-files set to true. This will print the number of open files the process can open on startup.

Alternatively, you can retrieve the max_file_descriptors for each node using the Nodes Info API, with:

curl localhost:9200/_nodes/process?pretty

您需要确保为运行ES的用户完成此操作，而不仅仅是root（除非您以root身份运行它）。

要做到这一点，我会遵循以下指示：（ http://elasticsearch-users.115913.n3.nabble.com/quot-Too-many-open-files-quot-error-on-Mac-OSX-td4034733.html ）。 假设你想要32k和64k作为限制：

In /etc/launchd.conf put:

limit maxfiles 32000 64000


Make sure in your ~/.bashrc file you are not setting the ulimit with something like "ulimit -n 1024".  

Open a new terminal, and run:

launchctl limit maxfiles
ulimit -a

进行这些更改后，请不要忘记重新启动。 然后当您在此命令行参数中启动elasticsearch pass时：

elasticsearch -XX:-MaxFDLimit

在我的Mac上执行上述步骤后，我从Elasticsearch获得以下响应：

curl http://localhost:9200/_nodes/process?pretty

{
  "cluster_name" : "elasticsearch",
  "nodes" : {
    "XXXXXXXXXXXXXXXXXXXXXXX" : {
      "name" : "Marrina Smallwood",
      "transport_address" : "inet[XX.XX.XX.XX:9300]",
      "host" : "MacBook-Pro-Retina.local",
      "ip" : "XX.XX.XX.XX",
      "version" : "1.1.1",
      "build" : "f1585f0",
      "http_address" : "inet[/XX.XX.XX.XX:9200]",
      "process" : {
        "refresh_interval" : 1000,
        "id" : 538,
        "max_file_descriptors" : 32768,
        "mlockall" : false
      }
    }
  }
}