![](/img/trans.png)
[英]Read from dynamic GCS bucket partitioned by date using Apache Beam and Dataflow
[英]Beam template for dataflow will not upload to bucket
我已經按照所有快速入門/文檔來創建基於 Java 的數據流模板。 我找到並完成的步驟是:
1)創建項目:
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.22.0 \
-DgroupId=com.example \
-DartifactId=JsonToBucket \
-Dversion="0.1" \
-Dpackage=com.example.dataflow \
-DinteractiveMode=false
2) mvn clean && mvn 編譯
這報告構建成功
3)編譯上傳到bucket
mvn compile exec:java -Pdataflow-runner \
-Dexec.mainClass=com.example.dataflow.JsonToBucket -Dexec.cleanupDaemonThreads=false \
-Dexec.args="--runner=DataflowRunner \
--project=PROJECT \
--stagingLocation=gs://some-bucket/staging \
--templateLocation=gs://some-bucket/beam_templates/DisruptivePubSubToBucket"
output 是:
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------------< com.example:jsonToBucket >----------------------
[INFO] Building jsonToBucket 0.1
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ jsonToBucket ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /somepath/jsonToBucket/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ jsonToBucket ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 4 source files to /somepath/jsonToBucket/target/classes
[INFO]
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ jsonToBucket ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.161 s
[INFO] Finished at: 2020-06-19T14:05:52+02:00
[INFO] ------------------------------------------------------------------------
沒有文件上傳到步驟 3 中設置的存儲桶。
有什么問題?
問題出在run()
方法上。 取自@gixen 評論:
問題是一件真正的小事,run() 方法從未被觸發
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.