[英]PigUnit with Hadoop 2.x
我正在嘗試將PigUnit與Hadoop 2.4.1一起使用。 我有一個非常簡單的Pig腳本,我想在PigTest中執行,但是當我運行這個測試時,我得到以下異常:
java.lang.InstantiationError: org.apache.hadoop.mapreduce.JobContext
at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.createJobContext(HadoopShims.java:66)
at org.apache.pig.backend.hadoop.executionengine.fetch.FetchPOStoreImpl.createStoreFunc(FetchPOStoreImpl.java:58)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:103)
at org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.init(FetchLauncher.java:121)
at org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.launchPig(FetchLauncher.java:78)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:275)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1367)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1352)
at org.apache.pig.PigServer.storeEx(PigServer.java:1011)
at org.apache.pig.PigServer.store(PigServer.java:974)
at org.apache.pig.PigServer.openIterator(PigServer.java:887)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:752)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:228)
at org.apache.pig.pigunit.pig.PigServer.registerScript(PigServer.java:55)
at org.apache.pig.pigunit.PigTest.registerScript(PigTest.java:170)
at org.apache.pig.pigunit.PigTest.assertOutput(PigTest.java:242)
at com.mongodb.hadoop.pig.PigTest.testSimplePigScript(PigTest.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
在Hadoop 2.4.1中,org.apache.hadoop.mapreduce.JobContext是一個接口,而不是一個類,所以在我看來,像PigUnit在某種程度上期待一個舊版本的Hadoop? 如果確實如此,我如何將PigUnit指向更新的Hadoop罐子?
事實證明,我使用了錯誤的豬的verison。 Pig在Maven上有多個工件,我需要在build.gradle
中提供一個分類build.gradle
,以便指定適用於Hadoop 2.x的Pig jar:
compile "org.apache.pig:pig:${pigVersion}:h2"
這篇文章幫助我解決了Gradle /依賴問題。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.