从 PySpark 库中引用 Spark (Scala) 库

Question

I will be building a Python library for PySpark clients. This library will be calling a Spark (Scala) library that I already built and have in production. As motivation (and, perhaps, sanity-check), the Python library they will be calling would look something like this:

from pyspark.sql.DataFrame import PyDataFrame

def process(python_data_frame):
    sc = python_data_frame.rdd.context
    sql_context = python_data_frame.sql_ctx    
    processed_scala_df = sc._jvm.com.mayonesa.ScalaClass.process(python_data_frame._jdf)

    return PyDataFrame(processed_scala_df, sql_context)

I would like to make importing/using this library as painless as possible to my PySpark customers. How would I reference my Scala project as a dependency to/within this Python library? I would like to avoid them having to add attributes (ie, --jars ) to the spark-submit command.

Answer 1

--jars or --packages is the typical way to go with 3rd party libraries (like yours).

If you want to make their experience of using your libraries less painful, you might want to wrap spark-submit command with all the extra parameters into a wrapper script, which would definitely make it much easier to call.