How to add jar file to Apache Storm EXTLIB in runtime / no restart?

Question

I want to add new logic in a new jar file to Apache storm, when it is running. The way to feed it with new jars is to copy them to its extlib. The issue is when ever storm is running, the jars in this library are locked by it and new jars are not loaded. It seems like it reads the jar classes in loading time and cannot re-read them in run time. Can anyone give me a hint about how to add new jars to Storm in run time?

Answer 1

The way people generally do this is to bundle the dependencies with your topology jar, eg using the maven-shade-plugin. That way you get to run whatever unit or integration tests you have with the new dependency version, before pushing it to production. I would prefer doing this if you're updating code.

If you're updating resource bundles, or you don't want to do what I described above, I believe you can use the blobstore functionality to replace jars (haven't tested it though). Storm allows you to upload blobs to the cluster, which can then be added as dependencies to topologies. If you upload your dependencies as blobs to the cluster, you can tell Storm to restart your workers when the blobs are updated.

You could upload your dependency to the blobstore with storm blobstore create --file your-plugin.jar --acl o::rwa --replication-factor 1 plugin1 .

You would then submit your topology by doing something like storm jar your-topology.jar com.yourcompany.Topology -c topology.blobstore.map='{"plugin1":{"localname":"plugin1-blob", "uncompress":false, "workerRestart":true}}'

When you want to update your-plugin.jar , you do storm blobstore update --file your-updated-plugin.jar plugin1 . This should cause Storm to upload the new version of the jar, followed by restarting all the topology workers.

See more documentation on the blob store at https://storm.apache.org/releases/2.0.0-SNAPSHOT/distcache-blobstore.html , and documentation on the topology.blobstore.map setting at https://github.com/apache/storm/blob/v2.0.0/storm-client/src/jvm/org/apache/storm/Config.java#L110