dataproc 重命名由火花写入 GCS 文件夹中的文件

Question

I am using Dataproc to implement spark jobs using Scala. The aim of my spark job is to read data in GCS make some transformations and then write result data under GCS. The files we got from spark write are PART-00 , I want to rename them, but I can't find any solution as the written files are under gcs and not hdfs. Any idea how to solve this please. thanks a lot.

Answer 1

在 Dataproc 集群上，您仍然可以像对 HDFS 一样对 GCS 运行相同的hadoop fs -mv命令，除了使用完整的“gs://bucket/filename...”路径。