Hibernate 使用 AWS OpenSearch 搜索：10 分钟后 403 禁止访问

Question

过去几周我一直在努力解决这个问题。 索引工作“完美”，直到它不工作。 通常有大量实体 (10k+)，在所有实体的大约 70% 时它开始抛出403响应。

设置

包（gradle）：

implementation group: 'org.hibernate.search', name: 'hibernate-search-engine', version: '6.1.7.Final'
implementation group: "org.hibernate.search", name: "hibernate-search-mapper-orm", version: "6.1.7.Final"
implementation group: "org.hibernate.search", name: "hibernate-search-backend-lucene", version: "6.1.7.Final"
implementation group: "org.hibernate.search", name: "hibernate-search-backend-elasticsearch", version: "6.1.7.Final"
implementation group: "org.hibernate.search", name: "hibernate-search-backend-elasticsearch-aws", version: "6.1.7.Final"

hibernate 配置：

cfg.setProperty("hibernate.search.backend.type", "elasticsearch")
cfg.setProperty("hibernate.search.backend.aws.signing.enabled", "true")
cfg.setProperty("hibernate.search.backend.aws.region",config.getProperty("AWS_REGION"))
cfg.setProperty("hibernate.search.backend.aws.credentials.type", "static")
cfg.setProperty(
    "hibernate.search.backend.aws.credentials.access_key_id",
    config.getProperty("AWS_ACCESS_KEY_ID")
)
cfg.setProperty(
    "hibernate.search.backend.aws.credentials.secret_access_key",
    config.getProperty("AWS_SECRET_ACCESS_KEY")
)
cfg.setProperty("hibernate.search.backend.uris", config.getProperty("AWS_ES_HOSTS"))
cfg.setProperty("hibernate.search.backend.version_check.enabled", "false")
cfg.setProperty("hibernate.search.backend.version",config.getProperty("AWS_ES_VERSION"))
cfg.setProperty("hibernate.search.schema_management.strategy", "create")
cfg.setProperty("hibernate.search.backend.indexing.max_bulk_size", "20")
cfg.setProperty("hibernate.search.backend.indexing.queue_size", "20")

代码：

CompletableFuture.runAsync {
    Search.session(database.factory.openSession())
        .massIndexer()
        .idFetchSize(20)
        .batchSizeToLoadObjects(10)
        .dropAndCreateSchemaOnStart(false)
        .purgeAllOnStart(false)
        .startAndWait()
}

output：

起初......它工作正常：

PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 75.74% [128.930679documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33458 entities in 259447 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 75.85% [128.958908 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33508 entities in 259708 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 75.97% [129.021820 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33558 entities in 260016 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 76.08% [129.061295 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33608 entities in 260383 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 76.19% [129.071411 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33658 entities in 260650 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 76.31% [129.131012 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33708 entities in 260961 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 76.42% [129.168732 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33758 entities in 261316 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 76.53% [129.184586 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33808 entities in 261771 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 76.65% [129.151047 documents/second].

然后......它不会：

ERROR LogFailureHandler:36 - HSEARCH000058: Exception occurred org.hibernate.search.util.common.SearchException: HSEARCH400588: Call to the bulk REST API failed: HSEARCH400007: Elasticsearch request failed: HSEARCH400090: Elasticsearch response indicates a failure.
Request: POST /_bulk with parameters {}
Response: 403 'Forbidden' from 'https://redacted.es.amazonaws.com' with body
"message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been"
}
Failing operation:
  Indexing instance of entity 'Entity' during mass indexing
Entities that could not be indexed correctly:
Entity#ac4650e6-2008-4fd5-949f-8541569d6ba9
org.hibernate.search.util.common.SearchException: HSEARCH400588: Call to the bulk REST API failed: HSEARCH400007: Elasticsearch request failed: HSEARCH400090: Elasticsearch response indicates a failure.
Request: POST /_bulk with parameters {}
Response: 403 'Forbidden' from 'https://redacted.es.amazonaws.com' with body
{
"message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been"
}
at org.hibernate.search.backend.elasticsearch.orchestration.impl.ElasticsearchDefaultWorkSequenceBuilder$BulkedWorkExecutionState.onBulkWorkComplete(ElasticsearchDefaultWorkSequenceBuilder.java:239) ~[hibernate-search-backend-elasticsearch-6.1.7.Final.jar:6.1.7.Final]
at org.hibernate.search.util.common.impl.Futures.lambda$handler$3(Futures.java:100) ~[hibernate-search-util-common-6.1.7.Final.jar:6.1.7.Final]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2144) ~[?:?]
at org.hibernate.search.backend.elasticsearch.client.impl.ElasticsearchClientImpl$1.onFailure(ElasticsearchClientImpl.java:127) ~[hibernate-search-backend-elasticsearch-6.1.7.Final.jar:6.1.7.Final]
at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:672) ~[elasticsearch-rest-client-7.17.4.jar:7.17.4]
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:408) ~[elasticsearch-rest-client-7.17.4.jar:7.17.4]
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:392) ~[elasticsearch-rest-client-7.17.4.jar:7.17.4]
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) ~[httpcore-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:182) ~[httpasyncclient-4.1.5.jar:4.1.5]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87) ~[httpasyncclient-4.1.5.jar:4.1.5]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40) ~[httpasyncclient-4.1.5.jar:4.1.5]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.15.jar:4.4.15]
at java.lang.Thread.run(Thread.java:830) ~[?:?]
Caused by: org.hibernate.search.util.common.SearchException: HSEARCH400007: Elasticsearch request failed: HSEARCH400090: Elasticsearch response indicates a failure.
Request: POST /_bulk with parameters {}
Response: 403 'Forbidden' from 'https://redacted.es.amazonaws.com' with body
{
"message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been"
}
at org.hibernate.search.backend.elasticsearch.work.impl.AbstractNonBulkableWork.handleResult(AbstractNonBulkableWork.java:84) ~[hibernate-search-backend-elasticsearch-6.1.7.Final.jar:6.1.7.Final]
at org.hibernate.search.backend.elasticsearch.work.impl.AbstractNonBulkableWork.lambda$execute$3(AbstractNonBulkableWork.java:66) ~[hibernate-search-backend-elasticsearch-6.1.7.Final.jar:6.1.7.Final]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642

我翻遍了我能找到的所有文档，尝试了不同的参数、偏移量、fetchId 大小等，但都无济于事。 我似乎无法挖掘的真正区别因素是它首先起作用（实体实际上被索引并显示），但它在一段时间后不起作用......（4-5 分钟）。 对此一无所获。

欢迎任何反馈、答案或问题。

Answer 1

您似乎遇到了 Hibernate 搜索的大量索引过程的问题。 索引最初似乎工作正常，但当实体数量增加时，在所有实体的大约 70% 时它开始失败并返回 403 响应。

此问题的原因可能是由于各种因素造成的，例如达到 Elasticsearch 的索引率限制或凭据问题。 您可以尝试增加“hibernate.search.backend.indexing.max_bulk_size”和“hibernate.search.backend.indexing.queue_size”属性以查看是否有帮助。

此外，您还可以尝试记录发送到 Elasticsearch 的请求，以获取有关该问题的更多信息。