Elasticsearch.Net 7 & Nest 7 - 无效的批量更新请求

Question

I'm trying to create a.Net Core (netcoreapp3.1) application to send my data from Azure Eventhub to ES 7.

I'm using the following packages:
ElasticSearch.Net 7.8.1
Nest 7.8.1

The data that I retrieve from Eventhub are 2 types that inherit from the IElasticBaseEntity.
I want to bulk update a list of these objects, which can be the object with all the information or the object to update one field of an already indexed person.
The field to match / search in ES is the id field.

To simplify my example I will use these dummy classes:

public interface IElasticBaseEntity
{
   string Id { get; set; }
   DateTime ServerTimestamp { get; set; }
}

The person class is the one with all the information

public abstract class Person: IElasticBaseEntity
{
   public string Id { get; set; }
   public string Firstname {get; set;}
   public string Name {get; set;}   
   public decimal? Score { get; set; }
}

The Score class is the update that I want to do on the indexed Person, based on the Id

public abstract class Score : IElasticBaseEntity
{
   public string Id {get; set;}
   public decimal? Score { get; set; }
}

I use this method to build the connection to ES

public static IElasticClient CreateInstance(ElasticsearchConfig config)
{
   IConnectionPool pool;    
   var connection = new HttpConnection();

   pool = new SniffingConnectionPool(new[] { new Uri(config.Url) }) { SniffedOnStartup = true };

   var connectionSettings = new ConnectionSettings(pool, connection);
   connectionSettings.BasicAuthentication(config.Username, config.Password);
   connectionSettings.ServerCertificateValidationCallback(ServerCertificateValidationCallback);
   connectionSettings.RequestTimeout(TimeSpan.FromMinutes(2)).EnableHttpCompression();
   var client = new ElasticClient(connectionSettings);
    
   return client;
}

So I started with the bulk command on ElasticClient. In the past I was able to add the objects using the descriptor.Index but of course, I want an update an not an insert/create of everything.

So I've come up with this, but for some reason I keep on receiving an error in Visual Studio 2019 on "Invalid /_bulk request" without any other information.

IEnumerable<IElasticBaseEntity> list = RetrievedData();

var descriptor = new BulkDescriptor();
            
foreach (var eachDoc in list)
{
   var doc = eachDoc;
   descriptor.Update<IElasticBaseEntity>(i => i
      .Id(doc.Id)
      .Doc(doc)
      .DocAsUpsert(true));   
}

var response = await _client.BulkAsync(descriptor);

// Try to debug & see what was send
if (response.ApiCall.RequestBodyInBytes != null)
{
   var jsonOutput = System.Text.Encoding.UTF8.GetString(response.ApiCall.RequestBodyInBytes);
}

The error that I receive is the following (retrieved from the response.DebugInformation ):

Invalid NEST response built from a unsuccessful () low level call on POST: /persons-20200717/_bulk

Invalid Bulk items:

Audit trail of this API call:

• [1] PingFailure: Node: https://myconnectiontoES:9243/ Exception: PipelineException Took: 00:00:01.2859155
• [2] SniffOnFail: Took: 00:00:02.1577638
• [3] SniffFailure: Node: https://myconnectiontoES:9243/ Exception: PipelineException Took: 00:00:02.0840985

OriginalException: Elasticsearch.Net.ElasticsearchClientException: Failed sniffing cluster state.. >Call: unknown resource

---> Elasticsearch.Net.PipelineException: Failed sniffing cluster state. ---> Elasticsearch.Net.PipelineException: An error occurred trying to read the response from the >specified node. at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken) --- End of inner exception stack trace --- at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken) at Elasticsearch.Net.RequestPipeline.SniffOnConnectionFailureAsync(CancellationToken >cancellationToken) at Elasticsearch.Net.Transport 1.PingAsync(IRequestPipeline pipeline, Node node, CancellationToken >cancellationToken) at Elasticsearch.Net.Transport 1.RequestAsync[TResponse](HttpMethod method, String path, >CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters) --- End of inner exception stack trace ---

Audit exception in step 1 PingFailure:

Elasticsearch.Net.PipelineException: An error occurred trying to read the response from the specified >node. at Elasticsearch.Net.RequestPipeline.PingAsync(Node node, CancellationToken cancellationToken)

Audit exception in step 3 SniffFailure:

Elasticsearch.Net.PipelineException: An error occurred trying to read the response from the specified >node. at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)

Request:

<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() >on ConnectionSettings to force it to be set on the response.>

Response:

So I looks like there is something wrong with my /_bulk request

What I've tried:

• Catch the /_bulk request with fiddler > the request is invalid & not send
• Try to set connectionSettings.DisableDirectStreaming(true); to print out the request > this is always null

So if anyone could point out the mistake I made in building the the /_bulk request or could point me in a direction to debug this & retrieve more information, I would be thankfull. Currently I'm going around in circles, re-reading the documentation, google-ing, but without any result.

Thanks

Answer 1

The failure is occurring when attempting to read the response to sniffing the cluster . It looks like sniffing has occurred because pinging the cluster has failed though, so it's worth checking that you can make a HEAD request to the base address of the Elasticsearch cluster using the credentials the client is configured to use.

Based on the port 9243 being used in the connection to Elasticsearch, I suspect the Elasticsearch cluster is running in Elastic Cloud . SniffingConnectionPool must not be used with an Elasticsearch cluster running in Elastic Cloud, as the addresses returned in the sniff response for where to reach Elasticsearch nodes are internal addresses which will be unreachable from the client. Instead, the CloudConnectionPool should be used, which has some convenience methods to use when creating an instance of ElasticClient

var client = new ElasticClient(
    "<cloud id retrieved from the Elastic Cloud web console>", 
    new BasicAuthenticationCredentials(config.Username, config.Password));

CloudConnectionPool will use http compression by default, but if you need more control over other configuration like request timeout, you can use

var pool = new CloudConnectionPool(
    "<cloud id retrieved from the Elastic Cloud web console>", 
    new BasicAuthenticationCredentials(config.Username, config.Password));
    
var settings = new ConnectionSettings(pool)
    .RequestTimeout(TimeSpan.FromMinutes(2));

var client = new ElasticClient(settings);

Answer 2

After going around in circles, I started creating test to check the basic ES command. Based on the example I found for ElasticSearch.Net, I was unable to succesfully run any of the examples.

So I checked the ES Index by trying it with a curl command.
curl -H "Content-Type: application/json" -u username:password -XPOST "https://elasticsearch_url:port/indexname/stats/1" -d "{\"application\": \"test\"}"

This command failed, with the message that I'm not allowed to contact ES. So we've checked the ES user and appearently there was an error in the role that was assigned to this user (in ElasticSearch). Once we fixed this, I was able to run my code & execute the partial update.

IEnumerable<IElasticBaseEntity> list = RetrievedData();

var descriptor = new BulkDescriptor();
        
foreach (var eachDoc in list)
{
   var doc = eachDoc;
   descriptor.Update<IElasticBaseEntity>(i => i
      .Id(doc.Id)
      .Doc(doc)
      .DocAsUpsert(true));   
}

var response = await _client.BulkAsync(descriptor);

The issue I have now, is that this runs for about 1 hour and then I get a SnifFailure. The connection to ES is retried but mostly ends in a "MaxTimeOutReached"

The RequestTimeOut is set to 2 minutes The Retries & timeout is not set, so the defaults are used (-> this is retrying until the RequestTimeout of 2 minutes is reached)

As you can see in the log below: I'm able to send docs to ES and then at some point I receive this error. Mostly after running my application for 1h.

2020-08-05 11:22:01.5553| INFO| 001 ES_indexName processed documents 08/05/2020 11:22:01 1 in 202.3803 ms
2020-08-05 11:29:28.9633| INFO| 001 ES_indexName processed documents 08/05/2020 11:29:28 1 in 179.7982 ms
2020-08-05 11:40:08.4666| INFO| 001 ES_indexName processed documents 08/05/2020 11:40:07 1 in 291.5695 ms
2020-08-05 11:47:26.2924|ERROR| failed Invalid NEST response built from a unsuccessful () low level call on POST: /ES_indexName/_bulk
# Invalid Bulk items:
# Audit trail of this API call:
 - [1] PingFailure: Node: https://ES_node1:port/ Exception: PipelineException Took: 00:01:40.0193513
 - [2] SniffOnFail: Took: 00:05:00.0132241
 - [3] SniffFailure: Node: https://ES_node2:port/ Exception: PipelineException Took: 00:01:40.0036955
 - [4] SniffFailure: Node: https://ES_node3:port/ Exception: PipelineException Took: 00:01:40.0019296
 - [5] SniffFailure: Node: https://ES_node1:port/ Exception: PipelineException Took: 00:01:40.0005639
 - [6] MaxTimeoutReached:
# OriginalException: Elasticsearch.Net.ElasticsearchClientException: Maximum timeout reached while retrying request. Call: unknown resource
 ---> Elasticsearch.Net.PipelineException: Failed sniffing cluster state.
 ---> System.AggregateException: One or more errors occurred. (An error occurred trying to write the request data to the specified node.) (An error occurred trying to write the request data to the specified node.) (An error occurred trying to write the request data to the specified node.)
 ---> Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)<---

 ---> (Inner Exception #2) Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)<---

   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
   at Elasticsearch.Net.RequestPipeline.SniffOnConnectionFailureAsync(CancellationToken cancellationToken)
   at Elasticsearch.Net.Transport`1.PingAsync(IRequestPipeline pipeline, Node node, CancellationToken cancellationToken)
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
# Audit exception in step 1 PingFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.PingAsync(Node node, CancellationToken cancellationToken)
# Audit exception in step 3 SniffFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
# Audit exception in step 4 SniffFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
# Audit exception in step 5 SniffFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
 400039.9975 ms MUST RETRY```