Elasticsearch的索引編制性能不佳

Question

目前，我正在使用彈性搜索來存儲和查詢一些日志。 我們建立了一個五節點彈性搜索集群。 其中兩個索引節點和三個查詢節點。 在索引節點中，兩台服務器上都有redis，logstash和elasticsearch。 elasticsearch使用NFS存儲作為數據存儲。 我們的要求是索引300個日志條目/秒。 但是我可以從elasticsearch獲得的最佳性能僅為每秒25個日志條目！ elasticsearch的XMX是16G。 每個組件的版本：

Redis: 2.8.12
logstash: 1.4.2
elasticsearch: 1.5.0

我們當前的索引設置如下：

     {
      "userlog" : {
        "settings" : {
          "index" : {
            "index" : {
              "store" : {
                "type" : "mmapfs"
              },
              "translog" : {
                "flush_threshold_ops" : "50000"
              }
            },
            "number_of_replicas" : "1",
            "translog" : {
              "flush_threshold_size" : "1G",
              "durability" : "async"
            },
            "merge" : {
              "scheduler" : {
                "max_thread_count" : "1"
              }
            },
            "indexing" : {
              "slowlog" : {
                "threshold" : {
                  "index" : {
                    "trace" : "2s",
                    "info" : "5s"
                  }
                }
              }
            },
            "memory" : {
              "index_buffer_size" : "3G"
            },
            "refresh_interval" : "30s",
            "version" : {
              "created" : "1050099"
            },
            "creation_date" : "1447730702943",
            "search" : {
              "slowlog" : {
                "threshold" : {
                  "fetch" : {
                    "debug" : "500ms"
                  },
                  "query" : {
                    "warn" : "10s",
                    "trace" : "1s"
                  }
                }
              }
            },
            "indices" : {
              "memory" : {
                "index_buffer_size" : "30%"
              }
            },
            "uuid" : "E1ttme3fSxKVD5kRHEr_MA",
            "index_currency" : "32",
            "number_of_shards" : "5"
          }
        }
      }
    }

這是我的logstash配置：

    input {
            redis {
                    host => "eanprduserreporedis01.eao.abn-iad.ea.com"
                    port => "6379"
                    type => "redis-input"
                    data_type => "list"
                    key => "userLog"
                    threads => 15
            }
        # Second reids block begin
            redis {
                    host => "eanprduserreporedis02.eao.abn-iad.ea.com"
                    port => "6379"
                    type => "redis-input"
                    data_type => "list"
                    key => "userLog"
                    threads => 15
            }
            # Second reids block end
    }

    output {
            elasticsearch {
                    cluster => "customizedlog_prod"
                    index => "userlog"
                    workers => 30
            }
           stdout{}
    }

一個非常奇怪的事情是，盡管當前索引速度僅為〜20 / s，但IO等待非常高，幾乎達到70％。 大部分是讀取流量。 通過nfsiostat，當前讀取速度約為200Mbps！ 因此，基本上，為每個日志條目建立索引，它將讀取大約10Mbits的數據，這很瘋狂，因為我們的日志條目的平均長度小於10K。 因此，我進行了彈性搜索的jstack轉儲，這是一個RUNNING線程的結果：

    "elasticsearch[somestupidhostname][bulk][T#3]" daemon prio=10 tid=0x00007f230c109800 nid=0x79f6 runnable [0x00007f1ba85f0000]
       java.lang.Thread.State: RUNNABLE
            at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
            at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)
            at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)
            at sun.nio.ch.IOUtil.read(IOUtil.java:197)
            at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:730)
            at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:715)
            at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
            at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
            at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
            at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
            at org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:221)
            at org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:152)
            at org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:506)
            at org.elasticsearch.common.lucene.uid.PerThreadIDAndVersionLookup.lookup(PerThreadIDAndVersionLookup.java:104)
            at org.elasticsearch.common.lucene.uid.Versions.loadDocIdAndVersion(Versions.java:150)
            at org.elasticsearch.common.lucene.uid.Versions.loadVersion(Versions.java:161)
            at org.elasticsearch.index.engine.InternalEngine.loadCurrentVersionFromIndex(InternalEngine.java:1002)
            at org.elasticsearch.index.engine.InternalEngine.innerCreate(InternalEngine.java:277)
            - locked <0x00000005fc76b938> (a java.lang.Object)
            at org.elasticsearch.index.engine.InternalEngine.create(InternalEngine.java:256)
            at org.elasticsearch.index.shard.IndexShard.create(IndexShard.java:455)
            at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:437)
            at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:149)
            at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:515)
            at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:422)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

誰能告訴我彈性搜索在做什么，為什么索引這么慢？ 並且有可能改善它嗎？

Answer 1

它可能不完全負責您的性能不佳，但請查看batch_size選項以獲取redis。 我敢打賭，如果您一次從Redis中提取1個以上的文檔，情況會更好。