类路径问题

Question

I am trying to run a map/reduce job and I am getting a java.lang.NoSuchMethodError. I did some research on this and this appears when my code is executed (not compiled). The correct version of the class and methods are there during compilation, but when trying to run, the correct method is not available. The jar file that is causing this is guava. I know this from the stack that is printed. I throws an error when trying to execute the following line of code:

ArrayDeque<Entry<String, String>> a = Queues.newArrayDeque();

This jar is part of the hadoop classpath because it comes with the CDH verson 5.3.0 that I am using. I have tried adding the correct version of guava to the classpath, but the error does not change. My questions are as follows:

1. I believe that I have correctly identified the issue. Does this seem reasonable to you? I have never come across this error before.

2. I believe that I need to remove the older version of guava from the classpath and add the new one. However, I really do not know where to begin with correcting this. The command that is issued to hadoop jar does not contain the older version of guava (in the -libjar parm). The jar is part of the hadoop classpath when I issue the command "hadoop classpath". So I am assuming that there is some hadoop config file I could edit to make this go away. Is that the correct way to go, or is there some other thing I need to do?

I am using Java 7, CDH 5.3.0, NetBeans 8.

TIA

Answer 1

At the time that I'm writing this, Hadoop has a dependency on Guava version 11.0.2. It uses the library pretty heavily in its internal implementation.

According to the Guava JavaDocs, the Queues#newArrayDeque method was added in version 12.0. If your code is compiling successfully, then that means that Guava version 12.0 or higher is available on your compilation classpath at build time, but since version 11.0.2 is supplied at runtime by Hadoop, the method doesn't exist, resulting in NoSuchMethodError .

Unfortunately, there is no reliable way to swap out a different Guava version in Hadoop. Specifically, I recommend that you do not attempt to replace the Guava 11.0.2 jar that ships in the Hadoop distro. Replacing this with a different Guava version is untested, and it would risk destabilizing the cluster.

The broader problem is that Hadoop's dependencies "leak" to its clients. HADOOP-11656 is an unimplemented feature request that would isolate Hadoop's internal dependencies away from clients, so that you could more easily use common libraries like Guava at your desired version. Meanwhile, until that feature is implemented, I think your only options are to stick to Guava 11.0.2 APIs, or possibly try inlining some of the Guava code that you really want into your own project directly. The code for Queues#newArrayDeque is visible on GitHub .

public static <E> ArrayDeque<E> newArrayDeque() {
  return new ArrayDeque<E>();
}

In this particular case, it looks like it will be easy to replace your code with a direct call to the java.util.ArrayDeque constructor. Thanks to the Java 7 diamond operator, it won't even be much more verbose.

ArrayDeque<Entry<String, String>> a = new java.util.ArrayDeque<>();