无法在Spark Kubernetes集群模式下读取本地文件

Question

在火花集群模式程序中读取存储在系统中的文件时遇到问题。这给我一个错误，提示“找不到文件”，但是文件存在于定义的位置。请提出一些建议，以便我可以读取本地使用kubernetes在Spark集群中添加文件。

Answer 1

You cannot refer local files on your machine when you submit Spark on Kubernetes.

The available solutions for your case might be:

• Use Resource staging server . Is not available in the main branch of Apache Spark codebase, so the whole integration is on your side.
• Put your file to the http/hdfs accessible location: refer docs
• Put your file inside Spark Docker image and refer it as local:///path/to/your-file.jar

If you are running local Kubernetes cluster like Minikube you can also create a Kubernetes Volume with files you are interested in and mount it to the Spark Pods: refer docs . Be sure to mount that volume to both Driver and Executors.