随着记录的增长，mongoldb 文档更新的性能下降

Question

我有一个 iOS 应用程序，它将批量数据发送到 API 端点，该端点将数据存储到 mongodb 数据库中。 我的数据建模如下：

{
"_id" : ObjectId,
"device_id" : Uuid,
"rtfb_status": bool,
"repetitions" : [
    {
        "session_id" : Uuid,
        "set_id" : Uuid,
        "level" : String,
        "exercise" : String,
        "number" : i32,
        "rom" : f64,
        "duration" : f64,
        "time" : i64
    },
    ...,
],
"imu_data": [
    {
        "session_id": Uuid,
        "data": [
            {
                "acc" : {
                    "y" : f64,
                    "z" : f64,
                    "x" : f64,
                    "time" : i64,
                },
                "gyro" : {
                    "y" : f64,
                    "z" : f64,
                    "x" : f64,
                    "time" : i64,
                }
            },
            ...,
        ]
    },
    ...,
]
}

我的应用程序只是附加到相关数组。

async fn append_to_list<S: Serialize + From<I>, I>(
    self,
    collection: Collection,
    source: I,
    field_name: &str,
) -> Result<i64, CollectionError> {
    let new_records =
        bson::to_bson(&S::from(source)).map_err(CollectionError::DbSerializationError)?;

    self.update_user_record(
        collection,
        bson::doc! { "$push": { field_name: new_records } },
    )
    .await
}

async fn update_user_record(
    self,
    collection: Collection,
    document: bson::Document,
) -> Result<i64, CollectionError> {
    let query = self.try_into()?;

    let update_options = mongodb::options::UpdateOptions::builder()
        .upsert(true)
        .build();

    let updated_res = collection
        .update_one(query, document, update_options)
        .await
        .map_err(CollectionError::DbError)?;

    Ok(updated_res.modified_count)
}

pub async fn add_imu_records(
    self,
    collection: Collection,
    imurecords: JsonImuRecordSet,
) -> Result<i64, CollectionError > {
    self.append_to_list::<ImuDataUpdate, _>(collection, imurecords, "imu_data")
        .await
}

一切正常，但随着时间的推移写入性能会下降。 从我的应用程序的记录器输出：

有小记录

 INFO  data_server > 127.0.0.1:50789 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 16.78034ms
 INFO  data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 7.737755ms
 INFO  data_server > 127.0.0.1:50817 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 7.143721ms
 INFO  data_server > 127.0.0.1:50789 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 5.021643ms
 INFO  data_server > 127.0.0.1:50818 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 7.644989ms
 INFO  data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 4.456604ms
 INFO  data_server > 127.0.0.1:50817 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 2.822192ms
 INFO  data_server > 127.0.0.1:50789 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 1.820112ms
 INFO  data_server > 127.0.0.1:50818 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 1.850234ms
 INFO  data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 1.801561ms
 INFO  data_server > 127.0.0.1:50789 "PUT /v1/add_imu_records HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 26.722725ms

注意： add_imu_records 调用是一个更大的有效负载，所以我希望它运行更长时间。

但是在相对较短的持续时间（可能是 10 分钟左右）之后，写入的时间要长得多

大约 10 分钟的数据后

INFO  data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 23.000502ms
INFO  data_server > 127.0.0.1:50818 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 23.23503ms
INFO  data_server > 127.0.0.1:50789 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 114.679434ms
INFO  data_server > 127.0.0.1:50817 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 143.392153ms
INFO  data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 65.101141ms
INFO  data_server > 127.0.0.1:50818 "PUT /v1/add_imu_records HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 117.456596ms

难道我做错了什么？ mongodb 只是错误的工具，我应该使用 RDBMS 吗？ 我有一个在 Postgres 上运行的分支，响应时间比最好的 mongo 时间慢，但它们仍然非常稳定。

基于 Postgres 的服务器日志

INFO  data_server > 172.17.0.1:54918 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 7.300945ms
 INFO  data_server > 172.17.0.1:54906 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 5.927394ms
 INFO  data_server > 172.17.0.1:54910 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 6.025674ms
 INFO  data_server > 172.17.0.1:54914 "PUT /v1/add_imu_records HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 45.430983ms
 INFO  data_server > 172.17.0.1:54906 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 11.442257ms
 INFO  data_server > 172.17.0.1:54910 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 6.875235ms

Mongo 在我的机器上的 docker 容器中运行。 根据Object.bsonsize文档是 4484480 字节（4.48448 Mb）。

Answer 1

为了更新文档，MongoDB 必须从磁盘获取整个文档（除非它已经在缓存中），然后在内存中对其进行变异，然后将其写回磁盘。 修改也会写入 oplog 以复制到辅助节点。

随着文档大小的增加，每个文档需要更长的时间，并且由于每个文档消耗的内存空间越来越大，缓存流失也会开始侵蚀无关查询的性能。

MongoDB 中的最大文档大小为 16MB。 如果此文档在 10 分钟后已经达到 4MB，则需要尽快将其拆分或截断。