[英]how to read data from mongodb block by block and write to posrgres
I have a mongo db running in production. 我有一个正在运行的mongo db。 I want to move the data from MongoDB to Postgres for some migration requirement. 我想将数据从MongoDB移至Postgres,以实现某些迁移要求。
Now comes to data part, I am planning to write one utility which will read data from MongoDB and write to Postgres. 现在谈到数据部分,我打算编写一个实用程序,该实用程序将从MongoDB中读取数据并写入Postgres。
Here I want to read all the data from mongo db (contains 240335 rows) to Postgres. 在这里,我想从mongo db(包含240335行)到Postgres读取所有数据。
I can not read entire data into memory in the application. 我无法将整个数据读到应用程序的内存中。 I want to read in batch of 10000 then write do some modification and write those to Postgres and then again read next 10000 again repeat this. 我想批量读取10000,然后写一些修改并将其写入Postgres,然后再次读取下一个10000,再重复一次。
How can I do this? 我怎样才能做到这一点?
I never did it, but I think, you can use cursors to upload records severally. 我从没做过,但是我认为您可以使用游标分别上传记录。 Problem - the solution will be inefficient. 问题-解决方案将效率低下。
Example 例
var myCursor = db.bios.find( );
var myDocument = myCursor.hasNext() ? myCursor.next() : null;
if (myDocument) {
var myName = myDocument.name;
print (tojson(myName));
//put record to db or add to batch, and upload if 1000 in collection
}
Maybe you can use the streams? 也许您可以使用流? I do not know if streams can be used in MongoDB. 我不知道是否可以在MongoDB中使用流。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.