将外部本机库（.so）和外部jar与Hadoop MapReduce结合使用

Question

I'm new to the hadoop / java world, so please be gentle and feel free to correct egregious errors. I'm trying to use native libraries compiled on my ubuntu machine running Hadoop locally (standalone mode). I'm also trying to use an external .jar in addition to the .jar I have compiled. I tried making a fatjar unsuccessfully and decided I would attempt to pass the external jar and native library to hadoop via the command line. The libraries are used in a custom record reader I created. I am able to run mapreduce jobs without external libraries via the hadoop command. I am also able to run this program in eclipse when I set the LD_LIBRARY_PATH class variable. I'm unsure of variables that would need to be set to run this job successfully in hadoop, so please tell me if some are necessary, though I have tried setting $HADOOP_CLASSPATH.

ie

./bin/hadoop jar ~/myjar/cdf-11-16.jar CdfInputDriver -libjars cdfjava.jar -files libcdf.so,libcdfNativeLibrary.so input output

I've tried accessing the jar and so files from my local and copying them to HDFS.

I get the following error from the job:

Exception in thread "main" java.lang.NoClassDefFoundError: gsfc/nssdc/cdf/CDFConstants
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:274)
    at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1844)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1809)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1903)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:174)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:490)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
    at CdfInputDriver.run(CdfInputDriver.java:45)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at CdfInputDriver.main(CdfInputDriver.java:50)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.ClassNotFoundException: gsfc.nssdc.cdf.CDFConstants
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 36 more

I've tried seeing if the files were loaded in cache with the following code and it "cache files:" prints as null:

public class CdfInputDriver extends Configured implements Tool{

    @Override
    public int run(String[] args) throws Exception {

        Job job = Job.getInstance(getConf());

        System.out.println("cache files:" + getConf().get("mapreduce.job.cache.files"));
        Path[] uris = job.getLocalCacheFiles();
        for(Path uri: uris){

              System.out.println(uri.toString());
              System.out.println(uri.getName());            

        } 
        job.setJarByClass(getClass());

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(LongWritable.class);

        job.setInputFormatClass(CdfInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        job.setMapperClass(CdfMapper.class);
        //job.setReducerClass(WordCount.IntSumReducer.class);

        return job.waitForCompletion(true) ? 0 : 1;
    }

    public static void main(String[] args)  throws Exception,
        InterruptedException, ClassNotFoundException {
        int exitCode = ToolRunner.run(new CdfInputDriver(), args);
        System.exit(exitCode);
    }
}

Also, I am just testing this to inevitably run the job on Amazon EMR. Would storing the .so and .jar on S3 and using a similar method theoretically work?

Appreciate any help!

Answer 1

I figured this out for people having issues with this. For my scenario, it was a combination of issues.

./bin/hadoop jar ~/myjar/cdf-11-16.jar CdfInputDriver -libjars cdfjava.jar -files libcdf.so,libcdfNativeLibrary.so input output

A couple things had thrown me in a loop. Here are some things I checked. If anyone has factual info on why these contributed to it working, it would be appreciated.

(for the linux newbie) If you are running hadoop using sudo make sure to include -E to include environment variables.

Make sure the third party .jar library is located on your masternode. (seemed to be necessary but haven't confirmed with documentation... maybe my enviro variables were incorrect otherwise)

I was able to run this using Amazon EMR. I uploaded .so files and .jars to s3, ssh'd into the master node of the cluster, installed s3cmd via http://blog.adaptovate.com/2013/06/installing-s3cmd-on-ec2-so-that-yum.html , copied cdf-11-16.jar (mapreduce jar) and cdfjava.jar (third party jar) to the masternode with s3cmd get, and ran the job. I was able to reference to .so files on S3.