使用NEST获取ElasticSearch批量队列状态

Question

I have a program that performs several bulk index operation on an ElasticSearch cluster. At some point, I start getting errors like this one (snipped):

RemoteTransportException[...][indices:data/write/bulk[s]]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 100) ...];

Is there a way I can verify the status of the bulk upload queue, ideally using NEST, so that I can slow down the client application in case I see that the queue on the server is getting full?

The NodesInfo method looks interesting, but I don't see how to access the information I need:

using Nest;
using System;

class Program {
    static void Main(string[] args) {
        ElasticClient client = new ElasticClient(new ConnectionSettings(new Uri("http://whatever:9200/")));
        var nodesInfoResponse = client.NodesInfo();
        if (nodesInfoResponse.IsValid) {
            foreach (var n in nodesInfoResponse.Nodes) {
                Console.WriteLine($"Node: {n.Key}");
                var bulk = n.Value.ThreadPool["bulk"];
                // ???
            }
        }
    }
}

Answer 1

You need to use NodesStats() and not NodesInfo() .

var nodesStatsResponse = client.NodesStats();
if (nodesStatsResponse.IsValid)
{
    foreach (var node in nodesStatsResponse.Nodes)
    {
        long bulkThreadPoolQueueSize = node.Value.ThreadPool["bulk"].Queue;
    }
}

UPDATE: The above query will bring in a lot of information than required. A highly optimized request for getting the same information is through the usage of _cat/thread_pool API. See below:

var catThreadPoolResponse = client.CatThreadPool(d => d.H("host", "bulk.queue"));
if (catThreadPoolResponse.IsValid)
{
    foreach (var record in catThreadPoolResponse.Records)
    {
        string nodeName = record.Host;
        long bulkThreadPoolQueueSize = int.Parse(record.Bulk.Queue);
        Console.WriteLine($"Node [{nodeName}] : BulkThreadPoolQueueSize [{bulkThreadPoolQueueSize}]");
    }
}