How to include dependencies with Maven in Storm topology jar for cluster deployment?

Question

I'm trying to deploy a Storm topology to a local standalone cluster. I'll include links to my project's Github repo so you can see my code...

I have got a simple Storm topology running in local mode, which I have tweaked to deploy to a Dockerized cluster ( simple configuration based on the recommended instructions ).

However, when I try to get it running, I get this error:

Exception in thread "main" java.lang.NoClassDefFoundError: backtype/storm/topology/IRichSpout 

It seemed from this question that I just need to include storm-core as a dependency in my Maven project when building the topology.jar .

So I adjusted my pom.xml file (I'm not using Eclipse or any IDE, I'm just running Maven from the command line) to include storm-core as a dependency, and to include the maven-assembly-plugin to package the jar-with-dependencies . But now when I try to run the cluster I get an error saying:

...
java.io.IOException: Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar
...

So what exactly should I be including as a bundled dependency? And what lines of code do I need to add to/remove from my pom.xml to make it work?

Answer 1

I have got it working.

The problem was that the Storm dependencies on the Docker cluster were filed under org.apache.storm , whereas the local build was using backtype.storm . Changing the references to org.apache.storm seems to have done the trick.

Answer 2

You should add the "provided" scope in the storm dependency on the pom.xml file.