使用mongoimport将数据从HDFS导入MongoDB

Question

I have a set of files on HDFS. Can I directly load these files into mongoDB (using mongoimport) without copying the files from HDFS to my hard disk.

Answer 1

Have you tried MongoInsertStorage?

You can simply load the dataset using pig and then use MongoInsertStorage to dump directly into Mongo. It internally launches a bunch of mappers that do exactly what is mentioned by 'David Gruzman's answer on this page. One of the advantages of this approach, is the parallelism and speed you achieve due to multiple mappers simultaneously inserting into the Mongo collection.

Here's a rough cut of what can be done with pig

REGISTER mongo-java-driver.jar  
REGISTER mongo-hadoop-core.jar
REGISTER mongo-hadoop-pig.jar

DEFINE MongoInsertStorage com.mongodb.hadoop.pig.MongoInsertStorage();

-- you need this here since multiple mappers could spawn with the same
-- data set and write duplicate records into the collection
SET mapreduce.reduce.speculative false

-- or some equivalent loader
BIG_DATA = LOAD '/the/path/to/your/data' using PigStorage('\t'); 
STORE BIG_DATA INTO 'mongodb://hostname:27017/db USING MongoInsertStorage('', '');

More information here https://github.com/mongodb/mongo-hadoop/tree/master/pig#inserting-directly-into-a-mongodb-collection

Answer 2

If we speak about big data I would look into scalable solutions.
We had similar case of serious data set (several terabytes) sitting in HDFS. This data, although with some transformation was to be loaded into Mongo.
What we did was to develop MapReduce Job which run over data and each mapper inserts its split of data into mongodb via API.

Answer 3

Are you storing CSV/JSON files in HDFS? If so, you just need some way of mapping them to your filesystem so you can point mongoimport to the file.

Alternatively mongoimport will take input from stdin unless a file is specified.

Answer 4

您可以在没有--file参数的情况下使用mongoimport，并从stdin加载：

hadoop fs -text /path/to/file/in/hdfs/*.csv | mongoimport ...