PySpark 在独立模式下执行作业但使用用户定义的模块？

Question

I have installed spark on some machine to use it in standalone cluster mode. So now I have some machines that have for each the same spark build version (Spark 2.4.0 build on hadoop 2.7+).

I want to use this cluster for parallel data analysis and my language of run is Python so I'm using Pyspark not Spark. I have created some modules of the operations to process the data and give it the form that I want.

However, I don't want to copy manually all this modules that I have created on every machine so I would like to know which option are in PySpark to pass the dependencies so that for every executor I'm sure that the modules are present?

I have thought of virtual environments that will be activated and install the modules but I don't know how to do it in Spark Standalone mode, while in YARN manager seems to be this option, but I won't install YARN.

Ps. Note: some module use data files like .txt and some dynamic libraries like .dll, .so and I want that they are passed to the executors to.

Answer 1

A good solution to distribute Spark and your modules is to use Docker Swarm (I hope you have experience with Docker).

Try to give a look at this repository, it was very useful for me at the time https://github.com/big-data-europe/docker-spark

It is a good base for distributing Spark. On top of that you can build your own modules. So you create your personal Docker Images to distribute in your Docker Hub and then easily distribute them in your cluster with Docker Swarm