在 javascript mvc 项目中批量插入 mongodb

Question

我需要为别人开发的项目（与我的硕士学位相关）做出贡献，而我无法联系到实际的开发人员。 我是 MongoDB 的新手。 在项目中，他通过insertmany function将数据插入MongoDB。 但是，现在我尝试插入的数据更大，并给了我下面的错误。 我指定了相关的代码范围。 如何批量插入？ 我试图将数据拼接成更小的 arrays，但后来我得到回调已经调用错误。 我应该批量插入或拼接数据，我应该怎么做？

谢谢。

错误：

2020-05-02T19:50:20+0200 <info> admin.js:521 () Error occured..on bulk pool saving... { MongoError: BSONObj size: 16972498 (0x102FAD2) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "pools"
    at Function.MongoError.create (C:\TopicBinder2-master\node_modules\mongodb-core\lib\error.js:31:11)
    at C:\TopicBinder2-master\node_modules\mongodb-core\lib\connection\pool.js:497:72
    at authenticateStragglers (C:\TopicBinder2-master\node_modules\mongodb-core\lib\connection\pool.js:443:16)
    at Connection.messageHandler (C:\TopicBinder2-master\node_modules\mongodb-core\lib\connection\pool.js:477:5)
    at Socket.<anonymous> (C:\TopicBinder2-master\node_modules\mongodb-core\lib\connection\connection.js:331:22)
    at Socket.emit (events.js:198:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at Socket.Readable.push (_stream_readable.js:224:10)
    at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
  name: 'MongoError',
  message:
   'BSONObj size: 16972498 (0x102FAD2) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "pools"',
  operationTime:
   Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1588441812 },
  ok: 0,
  errmsg:
   'BSONObj size: 16972498 (0x102FAD2) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "pools"',
  code: 10334,
  codeName: 'Location10334',
  '$clusterTime':
   { clusterTime:
      Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1588441812 },
     signature: { hash: [Binary], keyId: [Long] } } }
2020-05-02T19:50:20+0200 <error> admin.js:400 () Error creating pool { message: 'Error occured on bulk saving!' }

var Pools = (module.exports = mongoose.model("pools", poolSchema));
module.exports.createPoolItems = function (poolItems, callback) {
  populateUniqueId(poolItems);

  Pools.collection.insertMany(poolItems, { ordered: false }, callback);
};

function populateUniqueId(poolItems) {
  if (poolItems.length > 0) {
    poolItems.forEach((element) => {
      element.unique_id =
        element.project + "_" + element.topic_id + "_" + element.document_id;
    });
  }
}

Answer 1

此错误与您的文档超过 16 MB 的事实有关。 这是 MongoDB 的限制。 您会看到更多详细信息： MongoDB 限制和阈值

...
 errmsg:
   'BSONObj size: 16972498 (0x102FAD2) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "pools"',
...

您可以从以下链接中找到数据 model 示例和模式：

MongoDB 数据建模