连接到 EMR 的 SageMaker notebook 导入自定义 Python 模块

Question

I looked through similar questions but none of them solved my problem. I have a SageMaker notebook instance, opened a SparkMagic Pyspark notebook connected to a AWS EMR cluster. I have a SageMaker repo connected to this notebook as well called dsci-Python

Directory looks like:

/home/ec2-user/SageMaker/dsci-Python
/home/ec2-user/SageMaker/dsci-Python/pyspark_mle/datalake_data_object/SomeClass
/home/ec2-user/SageMaker/dsci-Python/Pyspark_playground.ipynb

There are __init__.py under both pyspark_mle and datalake_data_object directory and I have no problem importing them in other environments

when I'm running this code in Pyspark_playground.ipynb:

from pyspark_mle.datalake_data_object.SomeClass.SomeClass import Something

I got No module named 'pyspark_mle'

I think this is an environment path thing.

Answer 1

The repo is on your Notebook Instance, whereas the PySpark kernel is executing code on the EMR cluster.

To access these local modules on the EMR cluster, you can clone the repository on the EMR cluster.

Also, SparkMagic has a useful magic send_to_spark which can be used to send data from the Notebook locally to the Spark kernel. https://github.com/jupyter-incubator/sparkmagic/blob/master/examples/Send%20local%20data%20to%20Spark.ipynb