如何在 Oozie spark scala 作业中将文件从本地复制到 HDFS 目录？

Question

I am trying to copy some files from local path to hdfs with scala, and running it with oozie. The job is failing as it is not able to read files from local path. Is there a way to read local files in oozie?

Answer 1

it is not possible to copy/read local files by spark if it is running in cluster mode. Reason is, When Oozie submits Spark job in cluster mode, it is not necessary that YARN will allocate the same node (local node) as an executor. Suppose if you have limited executors and it allocates the same node then also not possible to access the same file by all other executors.

Then only possible solution I see is to keep all local files in the share directory which will accessible by all cluster nodes after it you can use the below commands to fire hdfs command using scala.

import org.apache.hadoop.fs
import org.apache.hadoop.fs._
val conf = new Configuration()

val fs = path.getFileSystem(conf)

val hdfspath = new Path("hdfs:///user/nikhil/test.csv")
val localpath = new Path("file:///home/cloudera/test/")

fs.copyToLocalFile(hdfspath,localpath)

Please find below link to get help in creating of Share directory just for reference.

https://www.tecmint.com/how-to-setup-nfs-server-in-linux/