零件中的备份/还原集合

Question

I am currently working on an AWS EC2 server and I scraped some data that I stored in a MongoDB collection. This is the only collection in my database.

Now I need to transfer this collection to my local machine to process it. My problem is that my the remaining disk space on the remote machine is insufficient to dump the whole collection. There is room for around 60% of the collection. I tried to use db.copy() and db.export() using the host name to directly copy on my local machine but it doesn't work because I'm not on a local network and I have some authentification issues, even with a ssh tunnel.

What I would like to do is to split my big collection into 2 smaller collections and dump each of them. Is it possible?

Thank you!

Answer 1

Your best option is to use mongodump to just take parts of the collection. It's also the best thing for a "bulk migration" of data so there are usage parts than can apply to working directly between hosts if you can change the networking setup between the hosts to allow this.

If you need to use mongodump on only part of the collection then the general case is to apply the --query option to select your output. There is no "limit" modifier to the ouput so instead you need to apply the "range query" operators, which are $lte and $gt repectively.

As a trivial example set, consider the following data:

{ "_id" : ObjectId("560e4a56a1a451fc8a37057f"), "list" : [ 1, 2, 3 ] }
{ "_id" : ObjectId("560e4a5ca1a451fc8a370580"), "list" : [ 1, 2 ] }
{ "_id" : ObjectId("560e4a62a1a451fc8a370581"), "list" : [ 1 ] }
{ "_id" : ObjectId("560e4a6ca1a451fc8a370582"), "list" : [ ] }

So the idea is to get the _id value at the "cut point(s)" that you want and construct range queries to only select the documents within those ranges. For this example we will just break outputs into groups of two.

So the first thing you want is the _id of the second document ( being in twos ), which you can retrieve by applying .skip() and .limit() within the mongo shell:

db.sample.find().sort({ "_id": 1 }).skip(1).limit(1)

That is just going to return the document:

{ "_id" : ObjectId("560e4a5ca1a451fc8a370580"), "list" : [ 1, 2 ] }

Which is done by skipping over n-1 documents to the number you want to export in this batch and then just output the last document.

The issed mongodump would then contain the range selector for $lte to just come up to that point:

mongodump -d test -c sample \
--query '{ "_id": { "$lte": { "$oid": "560e4a5ca1a451fc8a370580" } } }' \
--out part1

Note the $oid within the query. The mongodump and mongoimport tools both use the "strict" form described in MongoDB Extended JSON . Helper constructors like ObjectId() available to the shell are not "strictly JSON", and tools like mongodump ( or anything with the --query option ) just use JSON as input, so such data is instead represented in this form.

For your next point you want to get the next n documents you can fit in the dump. So you want to query for the next document cut-off by either skipping the n documents already output plus the number of documents to the next cut off point n-1 , or basically ( 2 + 2 -1 ) = 3 :

db.sample.find().sort({ "_id": 1 }).skip(3).limit(1)

Or even better, apply the range with $gt from the last cut-off you had:

db.sample.find({ "_id": { "$gt": ObjectId("560e4a5ca1a451fc8a370580") }}).skip(1).limit(1)

Either way gets you the next cut-off document:

{ "_id" : ObjectId("560e4a6ca1a451fc8a370582"), "list" : [ ] }

Then apply another range query on the dump, but this time using "both" the $gt and $lte operators:

mongodump -d test -c sample \
--query '{ "_id": { 
    "$gt": { "$oid": "560e4a5ca1a451fc8a370580" },
    "$lte": { "$oid": "560e4a6ca1a451fc8a370582" } }}' \
--out part2

As with each part you can take the data and move it over to the target host as you require. Note that in this form --out specifies a directory where the files

Note that there are options that can help here as well, such as :

--host - (ideally from mongorestore ) Which can allow you to run the whole process on another system. So for example you could run the following in your new target MongoDB instance to pipe data from the origin host directly into mongorestore on that system:

mongodump --host orighost -d test -c sample \
--query '{ "_id": { 
    "$gt": { "$oid": "560e4a5ca1a451fc8a370580" },
    "$lte": { "$oid": "560e4a6ca1a451fc8a370582" } }}' \
--out - \
| mongorestore -d newtest -c newsample --dir -

Noting the - denotes standard output/input repectively for each command.

--gzip - If you have MongoDB 3.2 on both hosts, then you can also take advantage of this option to compress/decompress either the data output or stream as in the pipe above. Combined with that piping option it would be the most effecient way to migrate data to the new target host.

As for mongorestore in general, however you apply it the data will happily rebuild the collection, even in parts. The general behaviour is as marked "Insert Only" , so different restores will "add" to a collection but never "overwrite" data with the same _id value.

Look at the options carefully. As expecially if your host systems are both on EC2 or even both within general cloud resources, then there really should be no reason why you could not pipe output from one directly to the other. All that would be required is a little firewall configuration at most between the allowed hosts.

But if at any rate you just want to backup "partial" data, then this is usually the way to go about doing it.

Of course, depending on your own setup and authentication needs, both commands are likely to require other options than those demonstrated here. The options here are just the "required" options in order to specify a "collection" from a "database" and filter with a "query".