MongoDb 4.2：对分片集合进行索引扫描的查询非常慢
[英]MongoDb 4.2: query with index scan on a sharded collection is very slow
问题描述
I am experiencing very slow performance when running the following query against a sharded collection (MongoDb 4.2.1):
在对分片集合（MongoDb 4.2.1）运行以下查询时，我遇到了非常慢的性能：
db.dc.find({
'hierarchy.company': 'PICCOLO_SRL',
"hierarchy.brand": 'PICCOLO_PICCOLO',
'dcheader.doc_date': {
$gte: 1569016800,
$lte: 1571695140
},
'dcheader.cardnr': '0461000402452'})
The query uses the following index, that is not the shard index:该查询使用以下索引，而不是分片索引：
"key" : {
"hierarchy.company" : 1,
"hierarchy.brand" : 1,
"dcheader.doc_date" : 1,
"dcheader.cardnr" : 1
}
The collection contains 4.5M docs and its size is 20.2GB.该集合包含 4.5M 文档，其大小为 20.2GB。 The shard index is hashed type and the shard is balanced.分片索引是散列类型，分片是平衡的。
The query execution plan is the following:查询执行计划如下：
{
"queryPlanner" : {
"mongosPlannerVersion" : 1,
"winningPlan" : {
"stage" : "SHARD_MERGE",
"shards" : [
{
"shardName" : "rsshard1",
"connectionString" : "rsshard1/kcc1prod.sipos.it:27020,kcc2prod.sipos.it:27020,kcc3prod.sipos.it:27020",
"serverInfo" : {
"host" : "kcc2prod.sipos.it",
"port" : 27020,
"version" : "4.2.1",
"gitVersion" : "edf6d45851c0b9ee15548f0f847df141764a317e"
},
"plannerVersion" : 1,
"namespace" : "kaneda.dc",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"dcheader.cardnr" : {
"$eq" : "0461000402452"
}
},
{
"hierarchy.brand" : {
"$eq" : "PICCOLO_PICCOLO"
}
},
{
"hierarchy.company" : {
"$eq" : "PICCOLO_SRL"
}
},
{
"dcheader.doc_date" : {
"$lte" : 1571695140
}
},
{
"dcheader.doc_date" : {
"$gte" : 1569016800
}
}
]
},
"winningPlan" : {
"stage" : "SHARDING_FILTER",
"inputStage" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"hierarchy.company" : 1,
"hierarchy.brand" : 1,
"dcheader.doc_date" : 1,
"dcheader.cardnr" : 1
},
"indexName" : "IndexCardnrDocDate",
"isMultiKey" : false,
"multiKeyPaths" : {
"hierarchy.company" : [ ],
"hierarchy.brand" : [ ],
"dcheader.doc_date" : [ ],
"dcheader.cardnr" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"hierarchy.company" : [
"[\"PICCOLO_SRL\", \"PICCOLO_SRL\"]"
],
"hierarchy.brand" : [
"[\"PICCOLO_PICCOLO\", \"PICCOLO_PICCOLO\"]"
],
"dcheader.doc_date" : [
"[1569016800.0, 1571695140.0]"
],
"dcheader.cardnr" : [
"[\"0461000402452\", \"0461000402452\"]"
]
}
}
}
},
"rejectedPlans" : [
{
"stage" : "SHARDING_FILTER",
"inputStage" : {
"stage" : "FETCH",
"filter" : {
"$and" : [
{
"dcheader.cardnr" : {
"$eq" : "0461000402452"
}
},
{
"dcheader.doc_date" : {
"$lte" : 1571695140
}
},
{
"dcheader.doc_date" : {
"$gte" : 1569016800
}
}
]
},
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"hierarchy.company" : 1,
"hierarchy.brand" : 1,
"hierarchy.zone" : 1,
"hierarchy.pdv" : 1,
"dcheader.idscontrino" : 1
},
"indexName" : "IndexIdScontrino",
"isMultiKey" : false,
"multiKeyPaths" : {
"hierarchy.company" : [ ],
"hierarchy.brand" : [ ],
"hierarchy.zone" : [ ],
"hierarchy.pdv" : [ ],
"dcheader.idscontrino" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"hierarchy.company" : [
"[\"PICCOLO_SRL\", \"PICCOLO_SRL\"]"
],
"hierarchy.brand" : [
"[\"PICCOLO_PICCOLO\", \"PICCOLO_PICCOLO\"]"
],
"hierarchy.zone" : [
"[MinKey, MaxKey]"
],
"hierarchy.pdv" : [
"[MinKey, MaxKey]"
],
"dcheader.idscontrino" : [
"[MinKey, MaxKey]"
]
}
}
}
},
{
"stage" : "SHARDING_FILTER",
"inputStage" : {
"stage" : "FETCH",
"filter" : {
"$and" : [
{
"dcheader.cardnr" : {
"$eq" : "0461000402452"
}
},
{
"dcheader.doc_date" : {
"$lte" : 1571695140
}
},
{
"dcheader.doc_date" : {
"$gte" : 1569016800
}
}
]
},
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"hierarchy.company" : 1,
"hierarchy.brand" : 1,
"dcheader.deptnr" : 1,
"dcheader.tillid" : 1,
"dcheader.numdocumentogestionale" : 1,
"dcheader.fiscalprinternr" : 1
},
"indexName" : "IndexDcDocGestionale",
"isMultiKey" : false,
"multiKeyPaths" : {
"hierarchy.company" : [ ],
"hierarchy.brand" : [ ],
"dcheader.deptnr" : [ ],
"dcheader.tillid" : [ ],
"dcheader.numdocumentogestionale" : [ ],
"dcheader.fiscalprinternr" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"hierarchy.company" : [
"[\"PICCOLO_SRL\", \"PICCOLO_SRL\"]"
],
"hierarchy.brand" : [
"[\"PICCOLO_PICCOLO\", \"PICCOLO_PICCOLO\"]"
],
"dcheader.deptnr" : [
"[MinKey, MaxKey]"
],
"dcheader.tillid" : [
"[MinKey, MaxKey]"
],
"dcheader.numdocumentogestionale" : [
"[MinKey, MaxKey]"
],
"dcheader.fiscalprinternr" : [
"[MinKey, MaxKey]"
]
}
}
}
}
]
},
{
"shardName" : "rsshard2",
"connectionString" : "rsshard2/kcc1prod.sipos.it:27021,kcc2prod.sipos.it:27021,kcc3prod.sipos.it:27021",
"serverInfo" : {
"host" : "kcc3prod.sipos.it",
"port" : 27021,
"version" : "4.2.1",
"gitVersion" : "edf6d45851c0b9ee15548f0f847df141764a317e"
},
"plannerVersion" : 1,
"namespace" : "kaneda.dc",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"dcheader.cardnr" : {
"$eq" : "0461000402452"
}
},
{
"hierarchy.brand" : {
"$eq" : "PICCOLO_PICCOLO"
}
},
{
"hierarchy.company" : {
"$eq" : "PICCOLO_SRL"
}
},
{
"dcheader.doc_date" : {
"$lte" : 1571695140
}
},
{
"dcheader.doc_date" : {
"$gte" : 1569016800
}
}
]
},
"winningPlan" : {
"stage" : "SHARDING_FILTER",
"inputStage" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"hierarchy.company" : 1,
"hierarchy.brand" : 1,
"dcheader.doc_date" : 1,
"dcheader.cardnr" : 1
},
"indexName" : "IndexCardnrDocDate",
"isMultiKey" : false,
"multiKeyPaths" : {
"hierarchy.company" : [ ],
"hierarchy.brand" : [ ],
"dcheader.doc_date" : [ ],
"dcheader.cardnr" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"hierarchy.company" : [
"[\"PICCOLO_SRL\", \"PICCOLO_SRL\"]"
],
"hierarchy.brand" : [
"[\"PICCOLO_PICCOLO\", \"PICCOLO_PICCOLO\"]"
],
"dcheader.doc_date" : [
"[1569016800.0, 1571695140.0]"
],
"dcheader.cardnr" : [
"[\"0461000402452\", \"0461000402452\"]"
]
}
}
}
},
"rejectedPlans" : [
{
"stage" : "SHARDING_FILTER",
"inputStage" : {
"stage" : "FETCH",
"filter" : {
"$and" : [
{
"dcheader.cardnr" : {
"$eq" : "0461000402452"
}
},
{
"dcheader.doc_date" : {
"$lte" : 1571695140
}
},
{
"dcheader.doc_date" : {
"$gte" : 1569016800
}
}
]
},
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"hierarchy.company" : 1,
"hierarchy.brand" : 1,
"hierarchy.zone" : 1,
"hierarchy.pdv" : 1,
"dcheader.idscontrino" : 1
},
"indexName" : "IndexIdScontrino",
"isMultiKey" : false,
"multiKeyPaths" : {
"hierarchy.company" : [ ],
"hierarchy.brand" : [ ],
"hierarchy.zone" : [ ],
"hierarchy.pdv" : [ ],
"dcheader.idscontrino" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"hierarchy.company" : [
"[\"PICCOLO_SRL\", \"PICCOLO_SRL\"]"
],
"hierarchy.brand" : [
"[\"PICCOLO_PICCOLO\", \"PICCOLO_PICCOLO\"]"
],
"hierarchy.zone" : [
"[MinKey, MaxKey]"
],
"hierarchy.pdv" : [
"[MinKey, MaxKey]"
],
"dcheader.idscontrino" : [
"[MinKey, MaxKey]"
]
}
}
}
},
{
"stage" : "SHARDING_FILTER",
"inputStage" : {
"stage" : "FETCH",
"filter" : {
"$and" : [
{
"dcheader.cardnr" : {
"$eq" : "0461000402452"
}
},
{
"dcheader.doc_date" : {
"$lte" : 1571695140
}
},
{
"dcheader.doc_date" : {
"$gte" : 1569016800
}
}
]
},
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"hierarchy.company" : 1,
"hierarchy.brand" : 1,
"dcheader.deptnr" : 1,
"dcheader.tillid" : 1,
"dcheader.numdocumentogestionale" : 1,
"dcheader.fiscalprinternr" : 1
},
"indexName" : "IndexDcDocGestionale",
"isMultiKey" : false,
"multiKeyPaths" : {
"hierarchy.company" : [ ],
"hierarchy.brand" : [ ],
"dcheader.deptnr" : [ ],
"dcheader.tillid" : [ ],
"dcheader.numdocumentogestionale" : [ ],
"dcheader.fiscalprinternr" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"hierarchy.company" : [
"[\"PICCOLO_SRL\", \"PICCOLO_SRL\"]"
],
"hierarchy.brand" : [
"[\"PICCOLO_PICCOLO\", \"PICCOLO_PICCOLO\"]"
],
"dcheader.deptnr" : [
"[MinKey, MaxKey]"
],
"dcheader.tillid" : [
"[MinKey, MaxKey]"
],
"dcheader.numdocumentogestionale" : [
"[MinKey, MaxKey]"
],
"dcheader.fiscalprinternr" : [
"[MinKey, MaxKey]"
]
}
}
}
}
]
}
]
}
},
"executionStats" : {
"nReturned" : 0,
"executionTimeMillis" : 170263,
"totalKeysExamined" : 555702,
"totalDocsExamined" : 0,
"executionStages" : {
"stage" : "SHARD_MERGE",
"nReturned" : 0,
"executionTimeMillis" : 170263,
"totalKeysExamined" : 555702,
"totalDocsExamined" : 0,
"totalChildMillis" : 255673,
"shards" : [
{
"shardName" : "rsshard1",
"executionSuccess" : true,
"executionStages" : {
"stage" : "SHARDING_FILTER",
"nReturned" : 0,
"executionTimeMillisEstimate" : 1447,
"works" : 202303,
"advanced" : 0,
"needTime" : 202301,
"needYield" : 0,
"saveState" : 6882,
"restoreState" : 6882,
"isEOF" : 1,
"chunkSkips" : 0,
"inputStage" : {
"stage" : "FETCH",
"nReturned" : 0,
"executionTimeMillisEstimate" : 1428,
"works" : 202302,
"advanced" : 0,
"needTime" : 202301,
"needYield" : 0,
"saveState" : 6882,
"restoreState" : 6882,
"isEOF" : 1,
"docsExamined" : 0,
"alreadyHasObj" : 0,
"inputStage" : {
"stage" : "IXSCAN",
"nReturned" : 0,
"executionTimeMillisEstimate" : 1428,
"works" : 202302,
"advanced" : 0,
"needTime" : 202301,
"needYield" : 0,
"saveState" : 6882,
"restoreState" : 6882,
"isEOF" : 1,
"keyPattern" : {
"hierarchy.company" : 1,
"hierarchy.brand" : 1,
"dcheader.doc_date" : 1,
"dcheader.cardnr" : 1
},
"indexName" : "IndexCardnrDocDate",
"isMultiKey" : false,
"multiKeyPaths" : {
"hierarchy.company" : [ ],
"hierarchy.brand" : [ ],
"dcheader.doc_date" : [ ],
"dcheader.cardnr" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"hierarchy.company" : [
"[\"PICCOLO_SRL\", \"PICCOLO_SRL\"]"
],
"hierarchy.brand" : [
"[\"PICCOLO_PICCOLO\", \"PICCOLO_PICCOLO\"]"
],
"dcheader.doc_date" : [
"[1569016800.0, 1571695140.0]"
],
"dcheader.cardnr" : [
"[\"0461000402452\", \"0461000402452\"]"
]
},
"keysExamined" : 202302,
"seeks" : 202302,
"dupsTested" : 0,
"dupsDropped" : 0,
"indexDef" : {
"indexName" : "IndexCardnrDocDate",
"isMultiKey" : false,
"multiKeyPaths" : {
"hierarchy.company" : [ ],
"hierarchy.brand" : [ ],
"dcheader.doc_date" : [ ],
"dcheader.cardnr" : [ ]
},
"keyPattern" : {
"hierarchy.company" : 1,
"hierarchy.brand" : 1,
"dcheader.doc_date" : 1,
"dcheader.cardnr" : 1
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"direction" : "forward"
}
}
}
}
},
{
"shardName" : "rsshard2",
"executionSuccess" : true,
"executionStages" : {
"stage" : "SHARDING_FILTER",
"nReturned" : 0,
"executionTimeMillisEstimate" : 1677,
"works" : 353401,
"advanced" : 0,
"needTime" : 353399,
"needYield" : 0,
"saveState" : 12612,
"restoreState" : 12612,
"isEOF" : 1,
"chunkSkips" : 0,
"inputStage" : {
"stage" : "FETCH",
"nReturned" : 0,
"executionTimeMillisEstimate" : 1660,
"works" : 353400,
"advanced" : 0,
"needTime" : 353399,
"needYield" : 0,
"saveState" : 12612,
"restoreState" : 12612,
"isEOF" : 1,
"docsExamined" : 0,
"alreadyHasObj" : 0,
"inputStage" : {
"stage" : "IXSCAN",
"nReturned" : 0,
"executionTimeMillisEstimate" : 1656,
"works" : 353400,
"advanced" : 0,
"needTime" : 353399,
"needYield" : 0,
"saveState" : 12612,
"restoreState" : 12612,
"isEOF" : 1,
"keyPattern" : {
"hierarchy.company" : 1,
"hierarchy.brand" : 1,
"dcheader.doc_date" : 1,
"dcheader.cardnr" : 1
},
"indexName" : "IndexCardnrDocDate",
"isMultiKey" : false,
"multiKeyPaths" : {
"hierarchy.company" : [ ],
"hierarchy.brand" : [ ],
"dcheader.doc_date" : [ ],
"dcheader.cardnr" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"hierarchy.company" : [
"[\"PICCOLO_SRL\", \"PICCOLO_SRL\"]"
],
"hierarchy.brand" : [
"[\"PICCOLO_PICCOLO\", \"PICCOLO_PICCOLO\"]"
],
"dcheader.doc_date" : [
"[1569016800.0, 1571695140.0]"
],
"dcheader.cardnr" : [
"[\"0461000402452\", \"0461000402452\"]"
]
},
"keysExamined" : 353400,
"seeks" : 353400,
"dupsTested" : 0,
"dupsDropped" : 0,
"indexDef" : {
"indexName" : "IndexCardnrDocDate",
"isMultiKey" : false,
"multiKeyPaths" : {
"hierarchy.company" : [ ],
"hierarchy.brand" : [ ],
"dcheader.doc_date" : [ ],
"dcheader.cardnr" : [ ]
},
"keyPattern" : {
"hierarchy.company" : 1,
"hierarchy.brand" : 1,
"dcheader.doc_date" : 1,
"dcheader.cardnr" : 1
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"direction" : "forward"
}
}
}
}
}
]
}
},
"ok" : 1,
"operationTime" : Timestamp(1571924213, 6),
"$clusterTime" : {
"clusterTime" : Timestamp(1571924213, 6),
"signature" : {
"hash" : BinData(0,"YOMe4acOH+iIfoHlsq/dCErNiZc="),
"keyId" : 6708447541475672066
}
}
} }
As you can see, it seems like the SHARD_MERGE stage takes about 168000 millis: can anyone explain why?如您所见，SHARD_MERGE 阶段似乎需要大约 168000 毫秒：谁能解释为什么？
Please note that I have never had such bad performances using a previuos MongoDb version (3.6).请注意，我使用之前的 MongoDb 版本（3.6）从未有过如此糟糕的表现。
Also I have not found detailed information about the SHARD_MERGE stage: which process does perform this task (mongod or mongos)?此外，我还没有找到有关 SHARD_MERGE 阶段的详细信息：哪个进程执行此任务（mongod 或 mongos）？ What happens during this stage?在这个阶段会发生什么？
My architecture is composed of three Centos 7 machines, each one hosts three mongod processes (shard1 replica set, shard2 replica set and config replica set) and a mongos process.我的架构由三台 Centos 7 台机器组成，每台托管三个 mongod 进程（shard1 副本集、shard2 副本集和配置副本集）和一个 mongos 进程。
2 个解决方案
解决方案1
0 2019-10-25 06:15:16
The stages shown in that explain plan, in order of execution are:该解释计划中显示的阶段按执行顺序为：
at mongod shard:在 mongod 分片：
IXSCAN - compare the query against the index values IXSCAN - 将查询与索引值进行比较
FETCH - retrieve full documents FETCH - 检索完整文档
SHARDING_FILTER - eliminate any documents not owned by the shard (ie orphans) SHARDING_FILTER - 消除任何不属于分片的文档（即孤儿）
at mongos:在 mongos：
SHARD_MERGE - wait for, and then combine the result sets from the shards SHARD_MERGE - 等待，然后合并来自分片的结果集
The large disparity between the execution time on each shard and the SHARD_MERGE stage would suggest that either one shard started much later than the other, something delayed the response, or perhaps there is some clock skew going on here.每个分片的执行时间与 SHARD_MERGE 阶段之间的巨大差异表明，任何一个分片的启动时间都比另一个分片晚得多，或者某些东西延迟了响应，或者这里可能存在一些时钟偏差。
解决方案2
0 已采纳 2020-01-14 07:29:55
You can try to change the index order (if possible) to this.您可以尝试将索引顺序（如果可能）更改为此。
"key" : {
"hierarchy.company" : 1,
"hierarchy.brand" : 1,
"dcheader.cardnr" : 1
"dcheader.doc_date" : 1,
}
As a general rule of thumb index fields should be in the following order.作为一般经验法则，索引字段应按以下顺序排列。
- Equality fields before Range fields Range 字段之前的相等字段
- Sort fields before Range fields在范围字段之前对字段进行排序
- Equality Fields before Sort fields排序字段之前的相等字段
In your case Range field (doc_date) is before the Equality Field (cardnr).在您的情况下，范围字段（doc_date）在平等字段（cardnr）之前。
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.
相关问题
MongoDB小集合查询非常慢 - MongoDB small collection Query very slow
MongoDB即使使用索引也发现查询非常慢 - MongoDB find query very slow even with index
mongodb分片集合索引信息存储在哪里？ - Where the mongodb sharded collection index info stored?
如何实际更改 MongoDB 分片集合的分片键 (4.2) - How to actually change the shard key for MongoDB Sharded collection (4.2)
无法在 MongoDB 中获取对分片集合的覆盖查询 - Cannot get a covered query for sharded collection in MongoDB
MongoDB对集合的了解非常缓慢 - MongoDB getmore on a collection is very slow
MongoDB分片收集不重新平衡 - MongoDB sharded collection not rebalancing
MongoDB：$in Operator in Sharded collection - MongoDB: $in Operator in Sharded collection
虽然设置了索引，但简单的MongoDB查询非常慢 - Simple MongoDB query very slow although index is set
在分片集合上添加新索引后，MongoDB写性能下降 - MongoDB write performance drops after adding new index on a sharded collection
相关标签