[英]MongoDB updating all records in a collection with the results of query from another collection
I have around 40k records to update and each record gets the data from querying another collection.我有大约 40k 条记录要更新,每条记录都从查询另一个集合中获取数据。
I have an existing query to do this, but it runs for more than an hour.我有一个现有的查询来执行此操作,但它运行了一个多小时。 It usually disconnects, then I rerun it again.
它通常断开连接,然后我再次运行它。
I think there is a better way to do this, I am just a noob with mongodb and this solution works but I am not happy with the execution speed.我认为有更好的方法可以做到这一点,我只是 mongodb 的菜鸟,这个解决方案有效,但我对执行速度不满意。
Maybe you have a better or much faster solution.也许您有更好或更快的解决方案。
To better illustrate the data, please see below:为了更好地说明数据,请参见下图:
accounts帐户
[
{
"_id": ObjectId("AC101"),
"emails":null,
"name":"Account 101",
...
},
{
"_id": ObjectId("AC102"),
"emails":null,
"name":"Account 102",
...
},
{
"_id": ObjectId("AC103"),
"emails":null,
"name":"Account 103",
...
},
...
]
account_contacts account_contacts
[
{
"_id": Object("ACC001"),
"account": {
"$ref" : "account",
"$id" : ObjectId("AC101")
},
"email":"acc001@test.com",
"name":"Contact 001",
...
},
{
"_id": Object("ACC002"),
"account": {
"$ref" : "account",
"$id" : ObjectId("AC102")
},
"email":"acc002@test.com",
"name":"Contact 002",
...
},
{
"_id": Object("ACC003"),
"account": {
"$ref" : "account",
"$id" : ObjectId("AC103")
},
"email":"acc003@test.com",
"name":"Contact 003",
...
},
{
"_id": Object("ACC004"),
"account": {
"$ref" : "account",
"$id" : ObjectId("AC103")
},
"email":"acc004@test.com",
"name":"Contact 004",
...
},
{
"_id": Object("ACC005"),
"account": {
"$ref" : "account",
"$id" : ObjectId("AC103")
},
"email":"acc005@test.com",
"name":"Contact 005",
...
},
...
]
Query:询问:
db.getCollection('accounts').find({ 'emails':{ $eq:null } }).forEach(p => {
const emails = [];
db.getCollection('account_contacts').find({"account.$id": p._id}).forEach(c => {
emails.push(c.email);
});
db.getCollection('accounts').updateOne({"_id": p._id}, {$set: {"emails": emails}});
});
I have a filter to get only the accounts with null emails
, so that if it gets a timeout error (1hr)... I just rerun the script and it will process those accounts with null emails.我有一个过滤器,只获取带有 null
emails
的帐户,因此如果它出现超时错误(1 小时)......我只需重新运行脚本,它就会处理带有 null 电子邮件的帐户。
Currently, I do not have any idea on how to improve the query... but I know it is not the best solution for this case since it takes more than an hour.目前,我对如何改进查询一无所知......但我知道这不是这种情况的最佳解决方案,因为它需要一个多小时。
Update:更新:
While I still cannot make the aggregate/lookUp approach work, I did tried to run the old script in mongo console, which I ran before and executes more than an hour in my ID... If you run it directly in the mongo console, it only takes 12-14 mins which is not bad.虽然我仍然无法使聚合/查找方法起作用,但我确实尝试在 mongo 控制台中运行旧脚本,我之前运行过它并在我的 ID 中执行了一个多小时......如果你直接在 mongo 控制台中运行它,它只需要 12-14 分钟,这还不错。
This is what I did for now, but I still want to convert my script to use aggregation.这就是我现在所做的,但我仍然想将我的脚本转换为使用聚合。
TIA TIA
Using MongoDB 4.2, you can avoid pulling the documents to the client side if you are willing to use a temporary collection.使用 MongoDB 4.2,如果您愿意使用临时集合,则可以避免将文档拉到客户端。
Use aggregation to match all of the documents with null email, extract just the _id and store it in a temporary collection.使用聚合将所有文档与 null email 匹配,仅提取 _id 并将其存储在临时集合中。 Note that if you have an index on
{emails:1, _id:1}
it will streamline this part.请注意,如果您在
{emails:1, _id:1}
上有一个索引,它将简化这部分。 You may want to procedurally generate the temporary collection name so it doesn't use the same name for successive runs.您可能希望以程序方式生成临时集合名称,这样它就不会为连续运行使用相同的名称。
db.accounts.aggregate([
{$match: {emails: null}},
{$project: {_id: 1}},
{$out: "temporary_null_email_collection"}
])
Then aggregate the temporary collection, lookup the email from the account_contacts collection, get rid of any extraneous fields, and merge the results back with the accounts collection.然后聚合临时集合,从 account_contacts 集合中查找 email,去掉任何无关字段,并将结果与 accounts 集合合并回来。
db.temporary_null_email_collection.aggregate([
{$lookup:{
from: "account_contacts",
localField: "_id",
foreignField: "$id", // verify this field name is correct
as: contacts
}},
{$project: {
_id: 1,
emails: "$contacts.emails"
}},
{$merge: "accounts"}
])
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.