简体繁体 English

使用Mongodb Hadoop Connector进行分片

[英]Sharding With Mongodb Hadoop Connector

原文 2014-10-29 03:12:33 3 1 mongodb/ hadoop/ mapreduce/ sharding

HI i have the same case as mentioned in this question Hadoop HDFS MapReduce output into MongoDb 嗨，我有这个问题提到的情况相同的Hadoop HDFS MapReduce输出到MongoDb

I just want to use Hadoop to do Map Reduce (MR) from Mongodb and then the outputs are to be written back into MongoDB as collections. 我只想使用Hadoop从Mongodb执行Map Reduce（MR），然后将输出作为集合写回到MongoDB。 Though the above question has not been solved yet, i have more issue with this scenario. 尽管上述问题尚未解决，但在这种情况下我还有更多问题。 Here i need the output of MR to be written into MongoDB shards and to be sharded correctly. 在这里，我需要将MR的输出写入MongoDB分片并正确分片。

My question, is it possible to do such scenario since even i did that MR using MongoDB (not Hadoop), i got the output not split correctly into sharded servers and got duplicated output on the result. 我的问题是，是否有可能发生这种情况，因为即使我使用MongoDB（而不是Hadoop）来执行MR，我也无法将输出正确地拆分为分片服务器，并且结果重复输出。 If it's possible how to do that? 如果可能的话，该怎么做？ Thanks in advance. 提前致谢。

1 个解决方案

Yes. 是。 You write to mongo as usual. 您照常写给mongo。 The fact that your mongo db is set to run on shards is a detail that is hidden from you. 您的mongo db设置为在分片上运行的事实是一个对您隐藏的细节。