Apache Spark 3 GPU cluster

Question

I'm very new to Apache Spark. Before I was experimenting with Dask , Ray , and Horovod which can easily create GPU clusters. I'm currently using Apache Spark 3.0 (which added NVIDIA GPU support) but having trouble with creating GPU clusters. I attemped to configure the spark-defaults.conf as follows:

spark.executor.resource.gpu.amount 1
#spark.task.resource.gpu.amount    0.25
#spark.executor.resource.gpu.discoveryScript ./getGpusResources.sh

The state on the dashboard states waiting . How do I create the GPU cluster?

Answer 1

After reviewing several hidden websites, I compiled the instructions to setting up the GPU cluster in Apache Spark 3.0 in the following blog: http://deeplearningyogi.com/ Please comment.

Thanks,

vinhdiesal