从简单的Java程序调用Giraph作业

Question

I am new to Giraph and Hadoop Yarn. Following Giraph's quick start leads me to run the example job from jar build from source from command line.

I want to run the job from simple java program. The question is inspired from previous similar MapReduce job question . Looking for similar answers with java's dependencies which would be needed for that.

I have yarn setup locally - needs have a way to feed job to that from java program.

It is evident: https://giraph.apache.org/apidocs/org/apache/giraph/job/GiraphJob.html that there must have a way for this - but I am finding it hard to find examples for it with Yarn.

Answer 1

Found a way to do this from GiraphRunner's source code:

@Test
public void testPageRank() throws IOException, ClassNotFoundException, InterruptedException {

    GiraphConfiguration giraphConf = new GiraphConfiguration(getConf());
    giraphConf.setWorkerConfiguration(1,1,100);
    GiraphConstants.SPLIT_MASTER_WORKER.set(giraphConf, false);

    giraphConf.setVertexInputFormatClass(JsonLongDoubleFloatDoubleVertexInputFormat.class);
    GiraphFileInputFormat.setVertexInputPath(giraphConf,
                                             new Path("/input/tiny-graph.txt"));
    giraphConf.setVertexOutputFormatClass(IdWithValueTextOutputFormat.class);

    giraphConf.setComputationClass(PageRankComputation.class);

    GiraphJob giraphJob = new GiraphJob(giraphConf, "page-rank");       

    FileOutputFormat.setOutputPath(giraphJob.getInternalJob(),
                                   new Path("/output/page-rank2"));
    giraphJob.run(true);
}

private Configuration getConf() {
    Configuration conf = new Configuration();
    conf.set("fs.defaultFS", "hdfs://localhost:9000");

    conf.set("yarn.resourcemanager.address", "localhost:8032");
    conf.set("yarn.resourcemanager.hostname", "localhost");

    // framework is now "yarn", should be defined like this in mapred-site.xm
    conf.set("mapreduce.framework.name", "yarn");
    return conf;
}