spring-xd部署解决方案？

Question

I would like to run Spring-XD in a distributed set up. The processing part generally takes place in any one of the available containers. If I have 100 containers, the manifest should be available in all the containers, since we do not know exactly to which container the job may be deployed. Keeping the manifest/code in all the containers is the only solution? Suggest a way so that the manifest is used only by the container where it is executed.

Answer 1

There are two different things that can be deployed:

you can deploy a stream of the form source | ... | sink source | ... | sink source | ... | sink together with a deployment manifest through the shell using stream create (...) . Deployment manifests allows to specify the type of nodes where the modules of the stream are expected to run.

You can deploy custom modules, that you have written yourself to the xd-installation into the directory xd/modules/source , xd/modules/sink , xd/modules/processor , etc.

Deployment manifests of jobs or streams are stored as persistent nodes within Zookeeper automatically when a stream is defined. The Zookeeper process only runs on a single or some of the machines depending on your setup. See this section of the Spring XD Guide:

http://docs.spring.io/spring-xd/docs/current/reference/html/#_zookeeper_overview

I would not worry about how the deployment manifest is distributed. This should simply work out of the box.

Then you can deploy your module to all nodes using a single command in the spring-xd shell by typing module upload --file ... .

See here: http://docs.spring.io/spring-xd/docs/current/reference/html/#install-the-source-module

You could also set up your workers to read your custom module definition from a shared file system. Yet this would introduce a single point of failure (the file system).