Elasticsearch.Net 7 & Nest 7 - 無效的批量更新請求

Question

我正在嘗試創建一個.Net Core (netcoreapp3.1) 應用程序來將我的數據從 Azure Eventhub 發送到 ES 7。

我正在使用以下軟件包：
ElasticSearch.Net 7.8.1
巢 7.8.1

我從 Eventhub 檢索的數據是從 IElasticBaseEntity 繼承的 2 種類型。
我想批量更新這些對象的列表，可以是包含所有信息的 object 或 object 以更新已索引人員的一個字段。
ES中匹配/搜索的字段是id字段。

為了簡化我的示例，我將使用這些虛擬類：

public interface IElasticBaseEntity
{
   string Id { get; set; }
   DateTime ServerTimestamp { get; set; }
}

class 人是擁有所有信息的人

public abstract class Person: IElasticBaseEntity
{
   public string Id { get; set; }
   public string Firstname {get; set;}
   public string Name {get; set;}   
   public decimal? Score { get; set; }
}

分數 class 是我想要對索引的人做的更新，基於 Id

public abstract class Score : IElasticBaseEntity
{
   public string Id {get; set;}
   public decimal? Score { get; set; }
}

我使用這種方法來建立到 ES 的連接

public static IElasticClient CreateInstance(ElasticsearchConfig config)
{
   IConnectionPool pool;    
   var connection = new HttpConnection();

   pool = new SniffingConnectionPool(new[] { new Uri(config.Url) }) { SniffedOnStartup = true };

   var connectionSettings = new ConnectionSettings(pool, connection);
   connectionSettings.BasicAuthentication(config.Username, config.Password);
   connectionSettings.ServerCertificateValidationCallback(ServerCertificateValidationCallback);
   connectionSettings.RequestTimeout(TimeSpan.FromMinutes(2)).EnableHttpCompression();
   var client = new ElasticClient(connectionSettings);
    
   return client;
}

所以我從 ElasticClient 上的批量命令開始。 過去我可以使用descriptor.Index添加對象，但當然，我想要更新而不是插入/創建所有內容。

所以我想出了這個，但由於某種原因，我一直在 Visual Studio 2019 中收到關於“無效 /_bulk 請求”的錯誤，而沒有任何其他信息。

IEnumerable<IElasticBaseEntity> list = RetrievedData();

var descriptor = new BulkDescriptor();
            
foreach (var eachDoc in list)
{
   var doc = eachDoc;
   descriptor.Update<IElasticBaseEntity>(i => i
      .Id(doc.Id)
      .Doc(doc)
      .DocAsUpsert(true));   
}

var response = await _client.BulkAsync(descriptor);

// Try to debug & see what was send
if (response.ApiCall.RequestBodyInBytes != null)
{
   var jsonOutput = System.Text.Encoding.UTF8.GetString(response.ApiCall.RequestBodyInBytes);
}

我收到的錯誤如下（從response.DebugInformation檢索）：

從 POST 上的不成功 () 低級別調用構建的無效 NEST 響應：/persons-20200717/_bulk

無效的批量項目：

此 API 調用的審計跟蹤：

• [1] PingFailure: 節點: https://myconnectiontoES:9243/ 異常: PipelineException Take: 00:00:01.2859155
• [2] SniffOnFail: 采取: 00:00:02.1577638
• [3] SniffFailure：節點：https://myconnectiontoES:9243/ 異常：PipelineException Take: 00:00:02.0840985

OriginalException: Elasticsearch.Net.ElasticsearchClientException: 嗅探集群失敗 state .. >調用：未知資源

---> Elasticsearch.Net.PipelineException：嗅探集群 state 失敗。 ---> Elasticsearch.Net.PipelineException：嘗試從 > 指定節點讀取響應時發生錯誤。 at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken) --- End of inner exception stack trace --- at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken) at Elasticsearch.Net.RequestPipeline.SniffOnConnectionFailureAsync(CancellationToken >cancellationToken) at Elasticsearch.Net.Transport 1.RequestAsync[TResponse](HttpMethod method, String path, >CancellationTokenRequestParameterscancellationToken) -- PostData 1.PingAsync(IRequestPipeline pipeline, Node node, CancellationToken >cancellationToken) at Elasticsearch.Net.Transport - 內部異常堆棧跟蹤結束 ---

步驟 1 PingFailure 中的審計異常：

Elasticsearch.Net.PipelineException：嘗試從指定 >node 讀取響應時發生錯誤。 在 Elasticsearch.Net.RequestPipeline.PingAsync（節點節點，CancellationToken cancelToken）

步驟 3 SniffFailure 中的審計異常：

Elasticsearch.Net.PipelineException：嘗試從指定 >node 讀取響應時發生錯誤。 在 Elasticsearch.Net.RequestPipeline.SniffAsync（CancellationToken cancelToken）

要求：

<請求 stream 未被序列化程序捕獲或已讀取完成。 設置 DisableDirectStreaming() >on ConnectionSettings 以強制在響應中設置它。>

回復：

所以我的/_bulk請求似乎有問題

我試過的：

• 用 fiddler 捕獲/_bulk請求 > 請求無效且未發送
• 嘗試設置connectionSettings.DisableDirectStreaming(true); 打印出請求 > 這總是null

因此，如果有人可以指出我在構建/_bulk請求時所犯的錯誤，或者可以指出我調試這個和檢索更多信息的方向，我將不勝感激。 目前我正在轉圈，重新閱讀文檔，谷歌搜索，但沒有任何結果。

謝謝

Answer 1

嘗試讀取對嗅探集群的響應時發生故障。 似乎發生了嗅探，因為 ping 集群失敗，因此值得檢查您是否可以使用客戶端配置使用的憑據向 Elasticsearch 集群的基地址發出HEAD請求。

根據用於連接 Elasticsearch 的端口 9243，我懷疑 Elasticsearch 集群正在Elastic Cloud中運行。 SniffingConnectionPool不得與在 Elastic Cloud 中運行的 Elasticsearch 集群一起使用，因為在嗅探響應中返回的用於到達 Elasticsearch 節點的地址是客戶端無法訪問的內部地址。 相反，應該使用CloudConnectionPool ，它有一些在創建ElasticClient實例時可以使用的便捷方法

var client = new ElasticClient(
    "<cloud id retrieved from the Elastic Cloud web console>", 
    new BasicAuthenticationCredentials(config.Username, config.Password));

CloudConnectionPool將默認使用 http 壓縮，但如果您需要更多地控制其他配置，例如請求超時，您可以使用

var pool = new CloudConnectionPool(
    "<cloud id retrieved from the Elastic Cloud web console>", 
    new BasicAuthenticationCredentials(config.Username, config.Password));
    
var settings = new ConnectionSettings(pool)
    .RequestTimeout(TimeSpan.FromMinutes(2));

var client = new ElasticClient(settings);

Answer 2

繞了一圈之后，我開始創建測試來檢查基本的 ES 命令。 根據我為 ElasticSearch.Net 找到的示例，我無法成功運行任何示例。

所以我通過使用 curl 命令嘗試檢查了 ES 索引。
curl -H "Content-Type: application/json" -u username:password -XPOST "https://elasticsearch_url:port/indexname/stats/1" -d "{\"application\": \"test\"}"

該命令失敗，消息是我不允許聯系 ES。 因此，我們檢查了 ES 用戶，顯然分配給該用戶的角色存在錯誤（在 ElasticSearch 中）。 一旦我們解決了這個問題，我就可以運行我的代碼並執行部分更新。

IEnumerable<IElasticBaseEntity> list = RetrievedData();

var descriptor = new BulkDescriptor();
        
foreach (var eachDoc in list)
{
   var doc = eachDoc;
   descriptor.Update<IElasticBaseEntity>(i => i
      .Id(doc.Id)
      .Doc(doc)
      .DocAsUpsert(true));   
}

var response = await _client.BulkAsync(descriptor);

我現在遇到的問題是，它運行了大約 1 個小時，然后我得到了 SnifFailure。 重試與 ES 的連接，但大多以“MaxTimeOutReached”結束

RequestTimeOut 設置為 2 分鍾重試和超時未設置，因此使用默認值（-> 這是重試，直到達到 2 分鍾的 RequestTimeout）

正如您在下面的日志中看到的那樣：我能夠將文檔發送到 ES，然后在某些時候我收到此錯誤。 主要是在運行我的應用程序 1 小時之后。

2020-08-05 11:22:01.5553| INFO| 001 ES_indexName processed documents 08/05/2020 11:22:01 1 in 202.3803 ms
2020-08-05 11:29:28.9633| INFO| 001 ES_indexName processed documents 08/05/2020 11:29:28 1 in 179.7982 ms
2020-08-05 11:40:08.4666| INFO| 001 ES_indexName processed documents 08/05/2020 11:40:07 1 in 291.5695 ms
2020-08-05 11:47:26.2924|ERROR| failed Invalid NEST response built from a unsuccessful () low level call on POST: /ES_indexName/_bulk
# Invalid Bulk items:
# Audit trail of this API call:
 - [1] PingFailure: Node: https://ES_node1:port/ Exception: PipelineException Took: 00:01:40.0193513
 - [2] SniffOnFail: Took: 00:05:00.0132241
 - [3] SniffFailure: Node: https://ES_node2:port/ Exception: PipelineException Took: 00:01:40.0036955
 - [4] SniffFailure: Node: https://ES_node3:port/ Exception: PipelineException Took: 00:01:40.0019296
 - [5] SniffFailure: Node: https://ES_node1:port/ Exception: PipelineException Took: 00:01:40.0005639
 - [6] MaxTimeoutReached:
# OriginalException: Elasticsearch.Net.ElasticsearchClientException: Maximum timeout reached while retrying request. Call: unknown resource
 ---> Elasticsearch.Net.PipelineException: Failed sniffing cluster state.
 ---> System.AggregateException: One or more errors occurred. (An error occurred trying to write the request data to the specified node.) (An error occurred trying to write the request data to the specified node.) (An error occurred trying to write the request data to the specified node.)
 ---> Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)<---

 ---> (Inner Exception #2) Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)<---

   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
   at Elasticsearch.Net.RequestPipeline.SniffOnConnectionFailureAsync(CancellationToken cancellationToken)
   at Elasticsearch.Net.Transport`1.PingAsync(IRequestPipeline pipeline, Node node, CancellationToken cancellationToken)
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
# Audit exception in step 1 PingFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.PingAsync(Node node, CancellationToken cancellationToken)
# Audit exception in step 3 SniffFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
# Audit exception in step 4 SniffFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
# Audit exception in step 5 SniffFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
 400039.9975 ms MUST RETRY```