简体繁体 English

如何分发Apache Flume 1.4

[英]How to distribute Apache Flume 1.4

原文 2013-07-29 18:53:24 1 1 apache/ flume

I looked at Apache Flume 1.4 documentation and its not clear about how to distribute the work accross nodes. 我查看了Apache Flume 1.4文档，但对如何跨节点分配工作不清楚。 I have to fetch data from multiple sources and multiple query terms for each source and need to poll them regularly. 我必须从多个来源获取数据，并为每个来源获取多个查询词，并且需要定期对其进行轮询。 Older versions of Flume for flume-master and flume-node(s). 适用于flume-master和flume-node的Flume的较旧版本。 Cloudera release notes says these concepts are replaced by Agents. Cloudera发行说明说，这些概念已由代理替换。 But how to distribute the agents? 但是如何分配代理商？

Should I manually create multiple configurations on each node and start agents on each of the node? 我是否应该在每个节点上手动创建多个配置并在每个节点上启动代理？ or can be done through any centralized mechanism? 还是可以通过任何集中机制来完成？

1 个解决方案

You can configure Flume agents in Cloudera Manager - see Cloudera documentation (section Adding Flume). 您可以在Cloudera Manager中配置Flume代理-请参阅Cloudera文档（添加Flume）。 If you don't use Cloudera, it is possible to solve this problem with general purpose configuration management tools such as Puppet - it seems to be more flexible, although requires more effort. 如果您不使用Cloudera，则可以使用通用配置管理工具（例如Puppet）来解决此问题-尽管需要更多工作，但它似乎更灵活。