How to distribute Apache Flume 1.4

Question

I looked at Apache Flume 1.4 documentation and its not clear about how to distribute the work accross nodes. I have to fetch data from multiple sources and multiple query terms for each source and need to poll them regularly. Older versions of Flume for flume-master and flume-node(s). Cloudera release notes says these concepts are replaced by Agents. But how to distribute the agents?

Should I manually create multiple configurations on each node and start agents on each of the node? or can be done through any centralized mechanism?

Answer 1

You can configure Flume agents in Cloudera Manager - see Cloudera documentation (section Adding Flume). If you don't use Cloudera, it is possible to solve this problem with general purpose configuration management tools such as Puppet - it seems to be more flexible, although requires more effort.