将 pyspark pandas_udf 与 AWS EMR 一起使用时出现“No module named 'pandas'”错误

Question

I ran the code for this site ( https://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html#co-grouped-map ) using zeppelin on AWS EMR.

%pyspark
import pandas as pd
from pyspark.sql.functions import pandas_udf, PandasUDFType
    df1 = spark.createDataFrame(
        [(20000101, 1, 1.0), (20000101, 2, 2.0), (20000102, 1, 3.0), (20000102, 2, 4.0)],
        ("time", "id", "v1"))

df2 = spark.createDataFrame(
    [(20000101, 1, "x"), (20000101, 2, "y")],
    ("time", "id", "v2"))

def asof_join(l, r):
    return pd.merge_asof(l, r, on="time", by="id")

df1.groupby("id").cogroup(df2.groupby("id")).applyInPandas(
    asof_join, schema="time int, id int, v1 double, v2 string").show()

and got the "ModuleNotFoundError: No module named 'pandas'" error at running last row. df1.groupby("id").cogroup(df2.groupby("id")).applyInPandas( asof_join, schema="time int, id int, v1 double, v2 string").show()

> pyspark.sql.utils.PythonException:   An exception was thrown from
> Python worker in the executor. The below is the Python worker
> stacktrace. Traceback (most recent call last):   File
> "/mnt/yarn/usercache/zeppelin/appcache/application_1765329837897_0004/container_1765329837897_0004_01_000026/pyspark.zip/pyspark/worker.py",
> line 589, in main
>     func, profiler, deserializer, serializer = read_udfs(pickleSer, infile, eval_type)   File
> "/mnt/yarn/usercache/zeppelin/appcache/application_1765329837897_0004/container_1765329837897_0004_01_000026/pyspark.zip/pyspark/worker.py",
> line 434, in read_udfs
>     arg_offsets, f = read_single_udf(pickleSer, infile, eval_type, runner_conf, udf_index=0)   File
> "/mnt/yarn/usercache/zeppelin/appcache/application_1765329837897_0004/container_1765329837897_0004_01_000026/pyspark.zip/pyspark/worker.py",
> line 254, in read_single_udf
>     f, return_type = read_command(pickleSer, infile)   File "/mnt/yarn/usercache/zeppelin/appcache/application_1765329837897_0004/container_1765329837897_0004_01_000026/pyspark.zip/pyspark/worker.py",
> line 74, in read_command
>     command = serializer._read_with_length(file)   File "/mnt/yarn/usercache/zeppelin/appcache/application_1765329837897_0004/container_1765329837897_0004_01_000026/pyspark.zip/pyspark/serializers.py",
> line 172, in _read_with_length
>     return self.loads(obj)   File "/mnt/yarn/usercache/zeppelin/appcache/application_1765329837897_0004/container_1765329837897_0004_01_000026/pyspark.zip/pyspark/serializers.py",
> line 458, in loads
>     return pickle.loads(obj, encoding=encoding)   File "/mnt/yarn/usercache/zeppelin/appcache/application_1765329837897_0004/container_1765329837897_0004_01_000026/pyspark.zip/pyspark/cloudpickle.py",
> line 1110, in subimport
>     __import__(name)
> ModuleNotFoundError: No module named 'pandas'

The version of the library you are using is as follows "pyspark 3.0.0 spark 3.0.0 pyarrow 0.15.1 zeppelin 0.9.0" and set the zeppelin.pyspark.python config property to python3

Since pandas was not installed in the original EMR environment, I installed it with the command "sudo python3 -m pip install pandas". I have confirmed that if I run the code "import pandas" on zeppelin, it runs fine.

However, when I use pandas_udf from pyspark, I get an error pandas cannot be found. Why is this? How can I do it correctly?

Answer 1

Writing "sudo python3 -m install pandas" to the shell script for bootsrap action solve this.