Java SparkContext error: java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator

Question

This is my first foray into java on Spark. The following error is happening when using either Spark 1.X (tried 1.5.0 ) or 2.X (tried 2.2.0 ), java 1.8 and with scala 2.10 :

JavaSparkContext sc = new JavaSparkContext(sparkConf);

Exception in thread "main" java.lang.NoSuchMethodError: 
io.netty.buffer.PooledByteBufAllocator.<init>(ZIIIIIII)V
    at org.apache.spark.network.util.NettyUtils.createPooledByteBufAllocator(NettyUtils.java:120)
    at org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:107)
    at org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:99)
    at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:70)
    at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:450)
    at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:56)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:246)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:432)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
    at KMeansMP.main(KMeansMP.java:38)

I had assumed it were a libraries mismatch but have been unable to isolate the exact incompatibility. Here are relevant portions of the pom.xml :

<properties>
    <spark.version>2.2.0</spark.version>
</properties>

..

<dependencies>
    <dependency>
        <groupId>org.apache.giraph</groupId>
        <artifactId>giraph-core</artifactId>
        <version>1.1.0-hadoop2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.7.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>${spark.version}</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.10</artifactId>
        <version>${spark.version}</version>
        <scope>compile</scope>
    </dependency>
</dependencies>

Any java sparkers out there with tips are encouraged to pitch in.

Answer 1

Both spark-core and giraph-core have dependency for netty-all . You need to exclude it from giraph-core .

<dependencies>
    <dependency>
        <groupId>org.apache.giraph</groupId>
        <artifactId>giraph-core</artifactId>
        <version>1.1.0-hadoop2</version>
        <exclusions>
            <exclusion>
                <groupId>io.netty</groupId>
                <artifactId>netty-all</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.7.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>${spark.version}</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.10</artifactId>
        <version>${spark.version}</version>
        <scope>compile</scope>
    </dependency>
</dependencies>