Java Apache Beam - 使用DataflowRunner保存文件“LOCALY”

Question

Can send the java code but currently, it's not necessary.

I have an issue as when I run the job as (DirectRunner - using Google VM Instance) it is working fine, as it saves information to the local file and carries on...

The problem appears when trying to use (DataflowRunner), and the error which I receive:

java.nio.file.NoSuchFileExtension: XXXX.csv
.....
.....
XXXX.csv could not be delete.

It could be deleted as it not even created.

Problem - how to write the file locally when running through DataflowRunner ??

PS Using Apache Beam

Pipeline (part of the code) - Reading from BigQuery and store data to Google storage (Special Character issue)

Answer 1

AFAIK when it is ran as a dataflow instance, you have to write file to GCS service (aka storage bucket) rather than local disk.

Did you try that already? to create storage bucket: https://cloud.google.com/storage/docs/creating-buckets