随着记录的增长，mongoldb 文档更新的性能下降
[英]Degrading performance of mongoldb document updates as record grows
问题描述
I have an iOS application which sends batches of data to an API endpoint which stores the data into a mongodb database.
我有一个 iOS 应用程序，它将批量数据发送到 API 端点，该端点将数据存储到 mongodb 数据库中。 My data is modeled like:我的数据建模如下：
{
"_id" : ObjectId,
"device_id" : Uuid,
"rtfb_status": bool,
"repetitions" : [
{
"session_id" : Uuid,
"set_id" : Uuid,
"level" : String,
"exercise" : String,
"number" : i32,
"rom" : f64,
"duration" : f64,
"time" : i64
},
...,
],
"imu_data": [
{
"session_id": Uuid,
"data": [
{
"acc" : {
"y" : f64,
"z" : f64,
"x" : f64,
"time" : i64,
},
"gyro" : {
"y" : f64,
"z" : f64,
"x" : f64,
"time" : i64,
}
},
...,
]
},
...,
]
}
My application just appends to the relevant array.我的应用程序只是附加到相关数组。
async fn append_to_list<S: Serialize + From<I>, I>(
self,
collection: Collection,
source: I,
field_name: &str,
) -> Result<i64, CollectionError> {
let new_records =
bson::to_bson(&S::from(source)).map_err(CollectionError::DbSerializationError)?;
self.update_user_record(
collection,
bson::doc! { "$push": { field_name: new_records } },
)
.await
}
async fn update_user_record(
self,
collection: Collection,
document: bson::Document,
) -> Result<i64, CollectionError> {
let query = self.try_into()?;
let update_options = mongodb::options::UpdateOptions::builder()
.upsert(true)
.build();
let updated_res = collection
.update_one(query, document, update_options)
.await
.map_err(CollectionError::DbError)?;
Ok(updated_res.modified_count)
}
pub async fn add_imu_records(
self,
collection: Collection,
imurecords: JsonImuRecordSet,
) -> Result<i64, CollectionError > {
self.append_to_list::<ImuDataUpdate, _>(collection, imurecords, "imu_data")
.await
}
Everything is working , but write performance drops off as time goes on.一切正常，但随着时间的推移写入性能会下降。 From the logger output of my application:从我的应用程序的记录器输出：
With small records有小记录
INFO data_server > 127.0.0.1:50789 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 16.78034ms
INFO data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 7.737755ms
INFO data_server > 127.0.0.1:50817 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 7.143721ms
INFO data_server > 127.0.0.1:50789 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 5.021643ms
INFO data_server > 127.0.0.1:50818 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 7.644989ms
INFO data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 4.456604ms
INFO data_server > 127.0.0.1:50817 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 2.822192ms
INFO data_server > 127.0.0.1:50789 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 1.820112ms
INFO data_server > 127.0.0.1:50818 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 1.850234ms
INFO data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 1.801561ms
INFO data_server > 127.0.0.1:50789 "PUT /v1/add_imu_records HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 26.722725ms
Note: The add_imu_records call is a much larger payload so I expect it to run longer.注意： add_imu_records 调用是一个更大的有效负载，所以我希望它运行更长时间。
But after a relatively short duration (maybe 10 minutes or so) the writes are taking MUCH longer但是在相对较短的持续时间（可能是 10 分钟左右）之后，写入的时间要长得多
After ~10 mins of data大约 10 分钟的数据后
INFO data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 23.000502ms
INFO data_server > 127.0.0.1:50818 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 23.23503ms
INFO data_server > 127.0.0.1:50789 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 114.679434ms
INFO data_server > 127.0.0.1:50817 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 143.392153ms
INFO data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 65.101141ms
INFO data_server > 127.0.0.1:50818 "PUT /v1/add_imu_records HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 117.456596ms
Am I doing something wrong?难道我做错了什么？ Is mongodb just the wrong tool and I should use an RDBMS? mongodb 只是错误的工具，我应该使用 RDBMS 吗？ I have a branch of this that runs on Postgres, and the response times are slower than the mongo times at their best, but they remain pretty stable.我有一个在 Postgres 上运行的分支，响应时间比最好的 mongo 时间慢，但它们仍然非常稳定。
Postgres based server log基于 Postgres 的服务器日志
INFO data_server > 172.17.0.1:54918 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 7.300945ms
INFO data_server > 172.17.0.1:54906 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 5.927394ms
INFO data_server > 172.17.0.1:54910 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 6.025674ms
INFO data_server > 172.17.0.1:54914 "PUT /v1/add_imu_records HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 45.430983ms
INFO data_server > 172.17.0.1:54906 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 11.442257ms
INFO data_server > 172.17.0.1:54910 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 6.875235ms
Mongo is running on my machine in a docker container. Mongo 在我的机器上的 docker 容器中运行。 According to Object.bsonsize the document is 4484480 bytes (4.48448 Mb).根据Object.bsonsize文档是 4484480 字节（4.48448 Mb）。
1 个解决方案
解决方案1
1 已采纳 2020-09-14 03:02:34
In order to update a document, MongoDB must fetch the entire document from disk (unless it is already in the cache), then mutate it in memory, and write it back to the disk.为了更新文档，MongoDB 必须从磁盘获取整个文档（除非它已经在缓存中），然后在内存中对其进行变异，然后将其写回磁盘。 The modifications are also written to the oplog for replication to the secondary nodes.修改也会写入 oplog 以复制到辅助节点。
As the document size grows, this will take longer for each document, and since each document consumes increasing space in memory, cache churn will also begin to eat away performance for unrelated queries.随着文档大小的增加，每个文档需要更长的时间，并且由于每个文档消耗的内存空间越来越大，缓存流失也会开始侵蚀无关查询的性能。
The maximum document size in MongoDB is 16MB. MongoDB 中的最大文档大小为 16MB。 If this document is already at 4MB after 10 minutes, it will need to be split or cutoff soon.如果此文档在 10 分钟后已经达到 4MB，则需要尽快将其拆分或截断。
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.