两个 Spark 结构化流作业无法写入相同的基本路径

Question

Spark Structured Streaming doesn't allow two structured streaming jobs to write data to the same base directory which is possible with using dstreams.

As _spark_metadata directory will be created by default for one job, second job cannot use the same directory as base path as already _spark_metadata directory is created by other job, It is throwing exception.

Is there any workaround for this, other than creating separate base path's for both the jobs.

Is it possible to create the _spark_metadata directory else where or disable without any data loss.

If I had to change the base path for both the jobs, then my whole framework will get impacted, So i don't want to do that.

Answer 1

No, changing the metadata directory name or location is not possible yet. You can refer to this link for more information.

Could you eleborate why you will have to change model of your project for changing a path. Is the path hardcoded? Or are you reading this data in a particular manner no that would be affected?

Edit 1: You can use Partitions here. Example, if the data is stored as parquet you can have partitions in you base path. You can add a column "src" that has source of data, like SW1 for Stream writer 1 and SW2 for Stream Writer 2.

These will have below paths in hdfs:

1. <base-path>/src=SW1
2. <base-path>/src=SW2

Now your each job can directly write to their corresponding partition and your other jobs can continue reading from base path.