Kafka主题与Kafka Connect合并到HDFS
[英]Kafka topic merging with Kafka Connect to HDFS
问题描述
Is it possible to configure Kafka Connect's HDFS connector to write/combine several separate topics into one file? 
是否可以配置Kafka Connect的HDFS连接器将多个单独的主题写入/组合到一个文件中？
The topics will contain messages with the same avro schema and I want KafkaConnect to act as an intermediary between those Kafka topics and HDFS. 主题将包含具有相同avro架构的消息，我希望KafkaConnect充当这些Kafka主题和HDFS之间的中介。 Worst case scenario the topic contents could be combined after being written to HDFS, but I feel like a cleaner and quicker way should be possible with the HDFS connector. 在最坏的情况下，主题内容可以在写入HDFS后进行组合，但我觉得使用HDFS连接器应该可以更清洁，更快捷。
1 个解决方案
解决方案1
3 已采纳 2016-09-01 00:11:05
Right now the HDFS connector will write each topic to its own directory. 现在，HDFS连接器会将每个主题写入其自己的目录。 You can combine directories in HDFS after writing, or combine topics in Kafka before writing to HDFS, but the connector itself will not do it. 您可以在写入后组合HDFS中的目录，或者在写入HDFS之前在Kafka中组合主题，但连接器本身不会这样做。
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.