使用Apache Flume将数据流式传输到HBase

Question

我正在尝试使用apache flume将数据加载到hbase中。 当我使用水槽将数据流传输到hadoop中时，它可以正常工作。 但是，当我启动水槽代理以将数据加载到hbase时，出现NoClassDefFoundError。

14/05/12 23:14:10 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:agent4.conf
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Added sinks: sink1 Agent: agent4
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent4]
14/05/12 23:14:10 INFO node.AbstractConfigurationProvider: Creating channels
14/05/12 23:14:10 INFO channel.DefaultChannelFactory: Creating instance of channel channel1 type FILE
14/05/12 23:14:10 INFO node.AbstractConfigurationProvider: Created channel channel1
14/05/12 23:14:10 INFO source.DefaultSourceFactory: Creating instance of source source1, type exec
14/05/12 23:14:10 INFO sink.DefaultSinkFactory: Creating instance of sink: sink1, type: org.apache.flume.sink.hbase.HBaseSink
14/05/12 23:14:10 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
    at org.apache.flume.sink.hbase.HBaseSink.<init>(HBaseSink.java:102)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at java.lang.Class.newInstance(Class.java:374)
    at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:43)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 17 more

这是我的水槽conf：

flume-env.sh

JAVA_HOME=/usr
FLUME_CLASSPATH=/home/alpha/apache-flume-1.4.0-bin/lib
HBASE_CLASSPATH=/home/alpha/hbase-0.98.1/lib
HBASE_HOME=/home/alpha/hbase-0.98.1
FLUME_HOME=/home/alpha/apache-flume-1.4.0-bin

agent4.conf

# Name the components on this agent
agent4.sources = source1
agent4.sinks = sink1
agent4.channels = channel1

# Describe/configure source1
agent4.sources.source1.type = exec
agent4.sources.source1.command = tail -f /tmp/testGenerate.csv

# Describe sink1
agent4.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
agent4.sinks.sink1.table = AdreamLumiHB
agent4.sinks.sink1.columnFamily =lumiCF
agent4.sinks.sink1.batchSize=5000
agent4.sinks.sink1.serializer.regex = ^(\d+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),.*
agent4.sinks.sink1.serializer.regexIgnoreCase = true
agent4.sinks.sink1.serializer.colNames = id,nom,valeur,batiment,etage,piece

# Use a channel which buffers events to a file
agent4.channels.channel1.type = FILE 
agent4.channels.channel1.transactionCapacity = 1000000 
agent4.channels.channel1.checkpointInterval 30000
agent4.channels.channel1.maxFileSize = 2146435071
agent4.channels.channel1.capacity 10000000 

# Bind the source and sink to the channel
agent4.sources.source1.channels = channel1
agent4.sinks.sink1.channel = channel1

Answer 1

将hbase类路径附加到flume时会发生什么？

FLUME_CLASSPATH=/home/alpha/apache-flume-1.4.0-bin/lib/\*:/home/alpha/hbase-0.98.1/lib/\*

注意：请勿在星号*之前添加反斜杠。 我将其放在此处是因为星号未显示在此编辑器上。

Answer 2

我建议将所有jar从hbase home的/ lib文件夹复制到flume的/ lib文件夹。 这帮助我解决了这个问题。