在风暴群集中的每台机器上严格分配一个特殊的螺栓

Question

I would like to start special bolt on every machine in my cluster.

Storm has pluggable scheduler, but documentation is very scanty. There is only one understandable example from xumingming. How could be implemented this pattern using IScheduler ?

Example

We have TCPBolt , which listens port 8765. TCPBolt parallelism == cluster size. We need to start one instance of this bolt on each machine.

Answer 1

The example and source code gives enough demo code to do this.

First you need to set the task number of TCPBolt to be the host number of your cluser.

Then in your IScheduler impl, using data in private Map<String, SchedulerAssignmentImpl> assignments; to check if the TCPBolt is assign to a host ( in SchedulerAssignmentImpl, using data Map<ExecutorDetails, WorkerSlot> executorToSlot; )

Finally, if not assigned, using the following code to assign

// re-get the aviableSlots
availableSlots = cluster.getAvailableSlots(specialSupervisor);
 
// since it is just a demo, to keep things simple, we assign all the
// executors into one slot.
cluster.assign(availableSlots.get(0), topology.getId(), executors);

This is just a rough idea. Hope it helps.