[英]How to use ParseJsons in Apache Beam / Google Dataflow?
[英]How to use Apache Beam DSL APIs?
我正在嘗試從Apache Beam文檔中實現DSL API示例。 我正在使用apache梁庫的最新版本(2.4.0)
我正在運行的代碼與docs中的相同:
@Rule
public final transient TestPipeline p = TestPipeline.create();
@Test
public void dslTest() {
RowType appType = RowSqlType
.builder()
.withIntegerField("appId")
.withVarcharField("description")
.withTimestampField("rowtime")
.build();
// Create a concrete row with that type.
Row row = Row.withRowType(appType)
.addValues(1, "Some cool app", new Date())
.build();
// Create a source PCollection containing only that row
PCollection<Row> testApps = PBegin
.in(p)
.apply(Create
.of(row)
.withCoder(appType.getRowCoder()));
PCollection<Row> filteredNames = testApps.apply(
BeamSql.query(
"SELECT appId, description, rowtime "
+ "FROM PCOLLECTION "
+ "WHERE appId=1"));
}
這總是失敗,並顯示以下錯誤:
java.lang.AssertionError
at org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:546)
at org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:365)
at org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:336)
at org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.convertToBeamRel(BeamQueryPlanner.java:165)
at org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.validateAndConvert(BeamQueryPlanner.java:156)
at org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.convertToBeamRel(BeamQueryPlanner.java:144)
at org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:73)
at org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:47)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:286)
at PipelineTest.dslTest(PipelineTest.java:42)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:324)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
進行此測試的正確方法是什么,或者這是一個錯誤並且dsl api無法正常工作?
看來這是一個已知問題,在計划者內部有一個斷言,Beam無法滿足,請參見此jira 。 目前尚未准備好解決方案。
當前的解決方法是禁用斷言,具體取決於您的構建系統:
如果您使用的是gradle,則在build.gradle中,您將看到以下內容:
\n 測試{\n jvmArgs“ -da”\n }\n
如果您使用的是maven,則在pom.xml中,您將看到以下內容:
\n\n <插件>\n <的groupId> org.apache.maven.plugins </的groupId>\n <artifactId的>行家-萬無一失-插件</ artifactId的>\n <結構>\n <argLine> -da </ argLine>\n\n
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.