Install spark 3.2 in SageMaker Notebook

Question

SageMaker supports pyspark 2.4 and I want to install latest version of pyspark in Sagemaker. Can you please let me know how can I install latest version of pyspark in SageMaker notebook.

Answer 1

I'm assuming you're referring to SageMaker notebook instances.

You can use the magic command %%local to run commands locally on your instance.

For example,

%%local
!pip install --upgrade pyspark

You may need to restart the kernel for the upgraded package, and you can test it like below -

%%local
import pyspark
print("PySpark version: ", pyspark.__version__)