使用 mongoose 和 node express 查找一个集合中的所有文档，更改数据并保存到另一个集合中

Question

I'm trying get select data from 1 collection, remove the fields I don't need, and add some new fields, and passing the data needed which is being replicated on each document for the new fields, and save them into a different collection.

Am I getting close or am I way off? Any help would be appreciated.

router.post('', (req, res, next) => {
    const taskDay = req.body.taskDay;
    switch (taskDay) {
        case 'taskSunday':
            MasterTask.find({taskSunday: true}).forEach(tempData => {
                const taskSheet = new Tasksheet({
                    taskDate: req.body.taskDate,
                    taskTicket: req.body.taskTicket,
                    taskTime: tempData.taskTime,
                    taskCompany: tempData.taskCompany,
                    taskDescription: tempData.taskDescription,
                    taskCompleted: req.body.taskCompleted,
                    taskComments: req.body.taskComments,
                    taskCompletedBy: req.body.taskCompletedBy,
                    taskTimeStamp: req.body.taskTimeStamp,
                    taskKnowledgeTitle: tempData.taskKnowledgeTitle,
                    taskKnowledgeUrl: tempData.taskKnowledgeUrl,
                    taskType: tempData.taskType
                }); 
                Tasksheet.insertMany(taskSheet).then(result => {
                    console.log(result);
                    res.status(200).json({message: 'task sheet created'})
                });
            });
            break;

Sample Data being pulled in find:

    _id: "5f0acb7209e5981d10bb765a",
    taskTime: "00:00",
    taskCompany: "CompanyName",
    taskDescription: "Some Needed Daily Task",
    taskSunday: true,
    taskMonday: true,
    taskTuesday: true,
    taskWednesday: true,
    taskThursday: true,
    taskFriday: true,
    taskSaturday: true,
    taskDisable: false,
    taskEndOfMonth: null,
    taskKnowledgeTitle: "Some Title",
    taskKnowledgeUrl: "Some URL",
    taskForSpecificDays: null,
    taskType: "Core",
    __v: 0
  }

Data transformation that I'm trying to save to new collection:

  {
    id: "mongooseID",
    taskTicket: "",
    taskDate: null,
    taskTime: "00:00",
    taskCompany: "somename",
    taskDescription: "This is an example description of a task.  This is going to be an extra long line of text to test.",
    taskCompleted: false,
    taskComments: "",
    taskCompletedBy: "",
    taskTimeStamp: "",
    taskKnowledgeTitle: "Test",
    taskKnowledgeUrl: "http://www.google.com",
    taskType: "Core"
  }

Answer 1

Try this.

MasterTask.find(
{
    taskSunday: true
},
function(err, tempData) {
    let taskSheet = [];
    tempData.forEach(t => {
        taskSheet.push({
            taskDate: req.body.taskDate,
            taskTicket: req.body.taskTicket,
            taskTime: t.taskTime,
            taskCompany: t.taskCompany,
            taskDescription: t.taskDescription,
            taskCompleted: req.body.taskCompleted,
            taskComments: req.body.taskComments,
            taskCompletedBy: req.body.taskCompletedBy,
            taskTimeStamp: req.body.taskTimeStamp,
            taskKnowledgeTitle: t.taskKnowledgeTitle,
            taskKnowledgeUrl: t.taskKnowledgeUrl,
            taskType: t.taskType
        })
    })

    Tasksheet.insertMany(taskSheet, (err, t) => {
        console.log(err)
        console.log(t)
    })
})

Answer 2

Well, you do it a bit wrong. If you want to simply re-export your collection values (without modifying it) you'd better use such utilities like mongodump or mongoexport (depends on scenario) and then import it viamongorestore or mongoimport

If you want to handle your problem with JS code, it's fine, but code above is a bit wrong, why is that? Because if you deal with real huge number of documents in a collection, this code will never be finished.

So it's better to use find query with a cursor, like this:

async function t () {
   try {
        await collection_name.find({taskSunday: true}).lean().cursor({batchSize: 10}).eachAsync(async (document) => {
                /**
                * Your could write any logic here related with your document
                */
                const taskSheet = new Tasksheet({
                    taskDate: document.taskDate,
                    taskTicket: document.taskTicket,
                    taskTime: document.taskTime,
                    taskCompany: document.taskCompany,
                    taskDescription: document.taskDescription,
                    taskCompleted: document.taskCompleted,
                    taskComments: document.taskComments,
                    taskCompletedBy: document.taskCompletedBy,
                    taskTimeStamp: document.taskTimeStamp,
                    taskKnowledgeTitle: document.taskKnowledgeTitle,
                    taskKnowledgeUrl: document.taskKnowledgeUrl,
                    taskType: document.taskType
                }); 
                taskSheet.save()
        }, { parallel: 10})
   } catch (e) {
       console.error(e)
   }
}

This code will be guaranteed finished, because it takes every 10 documents (it depends on {batchSize: 10} and { parallel: 10} values) from your original collection_name and insert in one-by-one in 10 streams. Will allows you not load all the collection in your RAM.

And also allows you to modify the necessary data on-the-fly