Conflicting .jars in my fat jar vs Hadoop

Question

I'd love to use the features of json-io 4.10.1. Unfortunately, my version of hadoop (2.8.4) bundles version 2.5.1. When my app runs, it pulls in json-io from /usr/lib/hadoop-yarn-lib instead of the classes bundled in my application .jar.

This newer version, for example, does not have the method JsonReader.jsonToJava with the second argument that accepts parameters, and this version does a better job of mapping my objects to/from json.

When executing the application, I get an error that the appropriate method could not be found. Ultimately as a stopgap, I removed the file /usr/lib/hadoop-yarn-lib/json-io-2.5.1.jar and the application found the "local" version and ran successfully.

So in my pom.xml, I declare json-io as a dependency:

<dependency>
  <groupId>com.cedarsoftware</groupId>
  <artifactId>json-io</artifactId>
  <version>4.10.1</version>
</dependency>

And I've configured the shade plugin to create a fat .jar. The resulting jar does contain JsonReader.class from the correct version of json-io.

This older jar is directly on hadoop's classpath (/usr/lib/hadoop-yarn-lib/*).

I expect the class loader to find the bundled JsonReader.class, but it's pulling the one from the classpath.

Answer 1

In Maven you can exclude deeper dependencies:

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-yarn</artifactId>
  <version>3.2.0</version>
  <exclusions>
    <exclusion>
      <groupId>com.cedarsoftware</groupId>
      <artifactId>json-io</artifactId>
    </exclusion>
  </exclusions>
</dependency>

This should prevent the pulling of the other json-io lib.