Data Locality in Spark on Kubernetes

Question

Do we need HDFS or S3 when running Spark on Kubernetes? Will data locality be that efficient if we use just the nfs storage type?
Or maybe there is something fundamentally wrong in my understanding of Spark on Kubernetes.

Answer 1

It depends. If you are working externally with data(HDFS/S3). Then you won't have data locality and performance won't be awesome.

You can run hdfs inside Kubernetes . To try and avoid this issue.