Deploy Storm topology remotely using "storm jar" command on Windows

Question

I am trying to deploy a storm topology to a remote machine using the storm jar command on Windows

storm jar jarName.jar mainClass

Here is what I have in my storm.yaml file:

storm.zookeeper.servers:
     - "HostName"

nimbus.host: "HostName"
nimbus.thrift.port: 6627

But whenever I try to deploy, it always tries to deploy to my local machine.

EDIT

I tried putting the following configurations but it does not work:

conf.put(Config.NIMBUS_HOST, "HostName");
conf.put(Config.NIMBUS_THRIFT_PORT,6627);
conf.put(Config.STORM_ZOOKEEPER_SERVERS,Arrays.asList(new String[]{"HostName"}));
conf.put(Config.STORM_ZOOKEEPER_PORT,2181);

I also tried the -c option on the command line as:

storm -c nimbus.host=HostName jar jarName.jar mainClass 

But it does not work; I get:

Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

I tried placing the -c part at different places but nothing seems to work.

Also, I get

The field Config.NIMBUS_HOST is deprecated

when I set:

conf.put(Config.NIMBUS_HOST, "HostName");

I am using Storm version 0.10.0-beta

EDIT2

Submitting works perfectly fine from IDE after I use

System.setProperty("storm.jar", "path to jar"); 

however, it does not work from command line.

Edit3

Thanks Matt for all your help. I was able to deploy the topology using the java command that eclipse executes. I am still unable to deploy using the storm jar command however. It complains that the topology already exists even though i dont see anything on the Storm UI. Also, when i deploy using Eclipse or the Java command from command line, the topology deploys successfully. How should i compare the difference between executing the java command and what the storm jar command is doing? i tried to echo the variables in this section of the cmd file

 if %storm-command% == jar (
    set STORM_OPTS=%STORM_CLIENT_OPTS% %STORM_OPTS% -Dstorm.jar=%2
    echo %STORM_OPTS%
    set CLASSPATH=%CLASSPATH%;%2

    echo %CLASSPATH%

    set CLASS=%3
    set args=%4
    goto start
    :start
    shift
    if [%4] == [] goto done
    set args=%args% %4

    echo args: %args%

    goto start

    :done
    echo %args%
    set storm-command-arguments=%args%

  )

but i dont see anything unusual. Also, i see the following exception in the stacktrace

    Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])

notice

 topologies:[]

and then it complains that the topology name exists

Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
        at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
        at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)

        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)

        at mainClass.submitTopology(mainClass.java:127)
        at mainClass.main(mainClass.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
        at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
        at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:613)
        at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:549)
        at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)

        at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16546)
        at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16531)
        at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
        at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)

        at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
        at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
        at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
        ... 5 more

finally, here is the entire stacktrace.

 X:\abc\xyz\target>ERROR StatusLogger Invalid URL C:/asd/ewt/apache-storm 0.10.0-beta/log4j2/cluster.xml java.net.MalformedURLException: unknown protocol: c
            at java.net.URL.<init>(URL.java:593)
            at java.net.URL.<init>(URL.java:483)
            at java.net.URL.<init>(URL.java:432)
            at java.net.URI.toURL(URI.java:1089)
            at org.apache.logging.log4j.core.config.ConfigurationFactory.getInputFromUri(ConfigurationFactory.java:296)
            at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:395)
            at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:254)
            at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:419)
            at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:138)
            at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:147)
            at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
            at org.apache.logging.log4j.LogManager.getContext(LogManager.java:175)
            at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:102)
            at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
            at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
            at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
            at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:277)
            at org.apache.log4j.Category.<init>(Category.java:56)
            at org.apache.log4j.Logger.<init>(Logger.java:35)
            at org.apache.log4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:59)
            at org.apache.log4j.Logger.getLogger(Logger.java:39)
            at mainClass.<clinit>(mainClass.java:32)
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field nimbus_uptime_secs' is unset!Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
                at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
                at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
                at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)

                at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)

                at mainClass.submitTopology(mainClass.java:127)
                at mainClass.main(mainClass.java:57)
        Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
                at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
                at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
        read(ClusterSummary.java:613)
                at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
        read(ClusterSummary.java:549)
                at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)

                at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
        resultStandardScheme.read(Nimbus.java:16546)
                at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
        resultStandardScheme.read(Nimbus.java:16531)
                at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
                at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)

                at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
                at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
                at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
                ... 5 more

and by the way what is that weird logging exception/error at the top of the stacktrace? could that have something to do with the weird behavior of storm jar command?

if i do

java -cp jarName.jar;C:\abc\asdasd\apache-storm-0.10.0-beta\lib\* mainClass

i get

513  [main] INFO  b.s.u.Utils - Using defaults.yaml from resources
582  [main] INFO  b.s.u.Utils - Using defaults.yaml from resources
590  [main] INFO  b.s.StormSubmitter - Generated ZooKeeper secret payload for MD
5-digest: -6100635753051151148:-8346710213172614614
591  [main] INFO  b.s.s.a.AuthUtils - Got AutoCreds []
599  [main] INFO  b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeM
s [2000] the maxSleepTimeMs [60000] the maxRetries [5]
626  [main] INFO  b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeM
s [2000] the maxSleepTimeMs [60000] the maxRetries [5]
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protoc
ol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName, uptime_secs:83775, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
        at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
        at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)

        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)

        at com.rbccm.th2015.storm.SampleTopologyX.submitTopology(SampleTopologyX.java:127)
        at com.rbccm.th2015.storm.SampleTopologyX.main(SampleTopologyX.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName,uptime_secs:83775, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
        at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
        at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:613)
        at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:549)
        at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)

        at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16546)
        at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16531)
        at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
        at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)

        at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
        at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
        at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
        ... 5 more

so its complaining about the same thing but via a different route. could this have something to do with my YAML? and what is this business about nimbus_uptime_secs?

Final Edit

ok, time to put this to rest. For now at least, it does not look like the sotrm jar command is suitable for windows. I can clearly see a difference in the command that storm.py tries to execute in Windows and the one that it tries to execute in Linux. Basically, the paths get screwed up in windows and that must be what's causing all of these weird exceptions.

Thanks Matt for all of your help and support. I really learned a lot.

Answer 1

Running a topology via storm jar does only work on Linux (but not on Windows). The "storm" command is a Python script that assumes to run on Linux (Window is officially not supported by Storm).

However, "storm" command only starts a regular java command with appropriate setting. If a topology is submitted via an IDE, the IDE does the same thing. If you use Eclipse, you can extract the used configuration as described in Eclipse: export running configuration

Pay attention, that you need to set JVM argument -Dstorm.jar=path\\to\\myJar.jar ; otherwise, StormSubmitter will not find the jar that should be submitted to Storm cluster.

After extracting the java command, it should allow you to submit a topology via command line on Windows. If extracting the command is not supported by your IDE, you can still put it together manually (even if this is a cumbersome solution).

It will be something like:

java -Dstorm.jar=yourJarFile.jar
     -classpath yourJarFile.jar:
                /home/mjsax/.m2/repository/org/apache/storm/storm-core/0.11.0-SNAPSHOT/storm-core-0.11.0-SNAPSHOT.jar:
                /home/mjsax/.m2/repository/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar:
                /home/mjsax/.m2/repository/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar:
                /home/mjsax/.m2/repository/org/ow2/asm/asm/4.0/asm-4.0.jar:
                /home/mjsax/.m2/repository/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar:
                /home/mjsax/.m2/repository/org/objenesis/objenesis/1.2/objenesis-1.2.jar:
                /home/mjsax/.m2/repository/org/clojure/clojure/1.6.0/clojure-1.6.0.jar:
                /home/mjsax/.m2/repository/clj-time/clj-time/0.8.0/clj-time-0.8.0.jar:
                /home/mjsax/.m2/repository/joda-time/joda-time/2.3/joda-time-2.3.jar:
                /home/mjsax/.m2/repository/compojure/compojure/1.1.3/compojure-1.1.3.jar:
                /home/mjsax/.m2/repository/org/clojure/core.incubator/0.1.0/core.incubator-0.1.0.jar:
                /home/mjsax/.m2/repository/org/clojure/tools.macro/0.1.0/tools.macro-0.1.0.jar:
                /home/mjsax/.m2/repository/clout/clout/1.0.1/clout-1.0.1.jar:
                /home/mjsax/.m2/repository/ring/ring-core/1.1.5/ring-core-1.1.5.jar:
                /home/mjsax/.m2/repository/commons-codec/commons-codec/1.6/commons-codec-1.6.jar:
                /home/mjsax/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:
                /home/mjsax/.m2/repository/commons-fileupload/commons-fileupload/1.2.1/commons-fileupload-1.2.1.jar:
                /home/mjsax/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:
                /home/mjsax/.m2/repository/hiccup/hiccup/0.3.6/hiccup-0.3.6.jar:
                /home/mjsax/.m2/repository/ring/ring-devel/1.3.0/ring-devel-1.3.0.jar:
                /home/mjsax/.m2/repository/clj-stacktrace/clj-stacktrace/0.2.7/clj-stacktrace-0.2.7.jar:
                /home/mjsax/.m2/repository/ns-tracker/ns-tracker/0.2.2/ns-tracker-0.2.2.jar:
                /home/mjsax/.m2/repository/org/clojure/tools.namespace/0.2.4/tools.namespace-0.2.4.jar:
                /home/mjsax/.m2/repository/org/clojure/java.classpath/0.2.2/java.classpath-0.2.2.jar:
                /home/mjsax/.m2/repository/ring/ring-jetty-adapter/1.3.0/ring-jetty-adapter-1.3.0.jar:
                /home/mjsax/.m2/repository/ring/ring-servlet/1.3.0/ring-servlet-1.3.0.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-server/7.6.13.v20130916/jetty-server-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/orbit/javax.servlet/2.5.0.v201103041518/javax.servlet-2.5.0.v201103041518.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-continuation/7.6.13.v20130916/jetty-continuation-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-http/7.6.13.v20130916/jetty-http-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-io/7.6.13.v20130916/jetty-io-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-util/7.6.13.v20130916/jetty-util-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/ring/ring-json/0.3.1/ring-json-0.3.1.jar:
                /home/mjsax/.m2/repository/cheshire/cheshire/5.3.1/cheshire-5.3.1.jar:
                /home/mjsax/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.3.1/jackson-core-2.3.1.jar:
                /home/mjsax/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.3.1/jackson-dataformat-smile-2.3.1.jar:
                /home/mjsax/.m2/repository/tigris/tigris/0.1.1/tigris-0.1.1.jar:
                /home/mjsax/.m2/repository/org/clojure/tools.logging/0.2.3/tools.logging-0.2.3.jar:
                /home/mjsax/.m2/repository/com/googlecode/disruptor/disruptor/2.10.4/disruptor-2.10.4.jar:
                /home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-api/2.1/log4j-api-2.1.jar:
                /home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-core/2.1/log4j-core-2.1.jar:
                /home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.1/log4j-slf4j-impl-2.1.jar:
                /home/mjsax/.m2/repository/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar:
                /home/mjsax/.m2/repository/org/slf4j/log4j-over-slf4j/1.6.6/log4j-over-slf4j-1.6.6.jar:
                /home/mjsax/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:
                /home/mjsax/.m2/repository/args4j/args4j/2.0.16/args4j-2.0.16.jar
    package.to.your.MainClass