Copy documents from one collection to another in MongoDB

Question

I have a problem but it seems to me that all the solutions I've seen are solving a problem more complex than mine.

I'm relatively new to MongoDB, but I've been using it with Apache Nutch to store crawled documents. As I am crawling in stages, and assessing each crawled result differently, to get the stats, I have been storing the results in different collections.

All I want to do is put these collections together into one big collection so I can translate and classify. All the fields are the SAME, so I don't need to add any new field.

This is the insert query I have been using for each collection:

new_crawl_130422_data.insert_one(
                        {
                        "database_url": proj_database_url,
                        "database_project_id": proj_database_id,
                        "projectname": proj_database_name,
                        "version": version,
                        "boost": boost,
                        "content": content,
                        "digest": digest,
                        "title": title,
                        "timestamp": timestamp,
                        "url": website,
                        "language": language

                        }

So, collection 1, and collection 2, have these same fields. Obviously, there would be multiple documents in each collection that have the same database_project_id as these are records mapped based on urls that match, and they have been transferred from Solr. Where the project name and url match, the document has been assigned the project_id, to show that it contains data for that project.

I thought it would be possible to just add on one collection to the end of the other like a list, but I have looked at these solutions - Copy few documents in a collection to another collection in mongo DB , Insert all documents from one collection into another collection in MongoDB database , Spring data mongodb- copy a collection and they all seem to be doing more complicated things than I need.

I don't want to change any collection, just create one large collection by putting two or more collections together.

I also saw this - https://www.mongodb.com/docs/v4.2/reference/method/db.collection.copyTo/ , but it is deprecated.

Thanks in advance.

Answer 1

If you like to copy all data then simply run:

db.new_crawl_130422_data.aggregate([
    {$out: "secondCollection"}
])

Note, all existing data in secondCollection will be deleted. If you like to add, ie merge the documents and keep existing data, then use:

db.new_crawl_130422_data.aggregate([
    {$merge: "secondCollection"}
])

If you have a large data set, then see How to copy a collection from one database to another in MongoDB