[英]Apache beam on google cloud dataflow
我正在嘗試在Google Cloud Dataflow上運行作業,但無法使部署正常工作。 使用DirectRunner可以正常運行該作業,但是一旦我切換到dataflow-runner,我就會收到以下異常:
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: No Runner was specified and the DirectRunner was not found on the classpath.
Specify a runner by either:
Explicitly specifying a runner by providing the 'runner' property
Adding the DirectRunner to the classpath
Calling 'PipelineOptions.setRunner(PipelineRunner)' directly
at org.apache.beam.sdk.options.PipelineOptions$DirectRunner.create(PipelineOptions.java:286)
at org.apache.beam.sdk.options.PipelineOptions$DirectRunner.create(PipelineOptions.java:276)
at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:575)
at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:516)
at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:155)
at org.apache.beam.sdk.options.PipelineOptionsValidator.validate(PipelineOptionsValidator.java:70)
at org.apache.beam.sdk.runners.PipelineRunner.fromOptions(PipelineRunner.java:44)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:138)
at my.package.SalesTransactions.main(SalesTransactions.java:218)
我的命令運行:
mvn compile exec:java -Dexec.mainClass=my.package.SalesTransactions -Dexec.args="--runner=DataflowRunner --project=my-project --tempLocation=gs://my-project/tmp" -Pdataflow-runner
發現我的錯誤。 從DirectRunner切換到DataFlowRunner時,我不得不在pom.xml中添加依賴項,而不僅僅是將其作為配置文件運行時依賴項。
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>${beam.version}</version>
</dependency>
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.