[英]flume doesn't work with hdfs
I seted flume with two nodes.I want to load data from the slave01 to hdfs. 我将flume设置为两个节点,我想将数据从slave01加载到hdfs。 slave01:example-conf.properties slave01:example-conf.properties
agent.sources = baksrc
agent.channels = memoryChannel
agent.sinks =avro-forward-sink
agent.sources.baksrc.type = exec
agent.sources.baksrc.command = tail -F /root/hadoo/test/data.txt
agent.sources.baksrc.checkperiodic = 1000
agent.sources.baksrc.channels =memoryChannel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.keep-alive = 30
agent.channels.memoryChannel.capacity = 10000
agent.channels.memoryChannel.transactionCapacity = 10000
agent.sinks.avro-forward-sink.type = avro
agent.sinks.avro-forward-sink.hostname = master
agent.sinks.avro-forward-sink.port = 23004
agent.sinks.avro-forward-sink.channel = memoryChannel
master: example-conf.properties 母版:example-conf.properties
agent.sources = avrosrc
agent.sinks =hdfs-write
agent.channels = memoryChannel
agent.sources.avrosrc.type =avro
agent.sources.avrosrc.bind =master
agent.sources.avrosrc.port =23004
agent.sources.avrosrc.channels=memoryChannel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.keep-alive = 30
agent.channels.memoryChannel.capacity = 10000
agent.channels.memoryChannel.transactionCapacity =10000
agent.sinks.hdfs-write.type = hdfs
agent.sinks.hdfs-write.hdfs.path =hdfs://172.16.86.38:9000/flume/webdata
agent.sinks.hdfs-write.hdfs.rollInterval = 0
agent.sinks.hdfs-write.hdfs.rollSize = 4000000
agent.sinks.hdfs-write.hdfs.rollCount = 0
agent.sinks.hdfs-write.hdfs.writeFormat = Text
agent.sinks.hdfs-write.hdfs.fileType = DataStream
agent.sinks.hdfs-write.hdfs.batchSize = 10
agent.sinks.hdfs-write.channel=memoryChannel
Then i run a shell script:like this: 然后我运行一个shell脚本:像这样:
#!/bin/sh
for i in {1..1000000}; do
echo "test flume to Hbase $i" >>/root/hadoop/test/data.txt;
sleep 0.1;
done
start flume: flume-ng agent --conf conf --conf-file example-conf.properties --name agent -Dflume.root.logger=DEBUG,console I got no error on the console. 启动flume:flume-ng代理--conf conf --conf-file example-conf.properties --name代理-Dflume.root.logger = DEBUG,控制台我在控制台上没有错误。
14/05/06 16:38:44 INFO source.AvroSource: Avro source avrosrc stopping: Avro source avrosrc: { bindAddress: master, port: 23004 }
14/05/06 16:38:44 INFO ipc.NettyServer: [id: 0x49f2de1b, /172.16.86.39:9359 :> /172.16.86.38:23004] DISCONNECTED
14/05/06 16:38:44 INFO ipc.NettyServer: [id: 0x49f2de1b, /172.16.86.39:9359 :> /172.16.86.38:23004] UNBOUND
14/05/06 16:38:44 INFO ipc.NettyServer: [id: 0x49f2de1b, /172.16.86.39:9359 :> /172.16.86.38:23004] CLOSED
but i can't see the file in hdfs, Is there any problem with my configuration ?? 但是我在hdfs中看不到文件,我的配置有问题吗? I've tested it on master,it work fine. 我已经在master上测试过了,工作正常。
which version of flume do you use ? 您使用哪个版本的水槽?
have you set HADOOP_HOME? 您设置了HADOOP_HOME吗?
does flume output the classpath with hadoop jar from HADOOP_HOME? flume是否从HADOOP_HOME用hadoop jar输出类路径?
if you are use apache flume,then step by step: 1. set HADOOP_HOME 如果您使用的是阿帕奇水槽,那么请逐步进行:1.设置HADOOP_HOME
2. edit hadoop core-site.xml,make sure namenode ip is right. 2.编辑hadoop core-site.xml,确保namenode ip正确。
3. use hdfs path: agent.sinks.hdfs-write.hdfs.path =/flume/webdata
3.使用hdfs路径: agent.sinks.hdfs-write.hdfs.path =/flume/webdata
4. start flume 4.开始水槽
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.