使用带有 pyspark 的 Jupyter Notebook 错误 no import named numpy

Question

As explained, I'm using pyspark in jupyter notebook. I'm getting the attached errors.

I have a tf-idf; I normalize it; then this last step creates a cosine-similarity matrix for documents.

from pyspark.mllib.linalg.distributed import IndexedRow, IndexedRowMatrix

mat = IndexedRowMatrix(
data.select("V2", "norm")\
    .rdd.map(lambda row: IndexedRow(row.ID, row.norm.toArray()))).toBlockMatrix()

But this is the error I'm getting: at the bottom it says "no module named numpy"

2022-08-25 15:16:26,161 WARN scheduler.DAGScheduler: Broadcasting large task binary with size 4.0 MiB
2022-08-25 15:16:27,561 ERROR executor.Executor: Exception in task 0.0 in stage 8.0 (TID 15)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/usr/local/Cellar/apache-spark/3.2.1/libexec/python/lib/pyspark.zip/pyspark/worker.py", line 601, in main
func, profiler, deserializer, serializer = read_command(pickleSer, infile)
File "/usr/local/Cellar/apache-spark/3.2.1/libexec/python/lib/pyspark.zip/pyspark/worker.py", line 71, in read_command
command = serializer._read_with_length(file)
File "/usr/local/Cellar/apache-spark/3.2.1/libexec/python/lib/pyspark.zip/pyspark/serializers.py", line 160, in _read_with_length
return self.loads(obj)
File "/usr/local/Cellar/apache-spark/3.2.1/libexec/python/lib/pyspark.zip/pyspark/serializers.py", line 430, in loads
return pickle.loads(obj, encoding=encoding)
File "<frozen zipimport>", line 259, in load_module
File "/usr/local/Cellar/apache-spark/3.2.1/libexec/python/lib/pyspark.zip/pyspark/mllib/__init__.py", line 26, in <module>
  import numpy
ModuleNotFoundError: No module named 'numpy'

at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:555)
at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:713)
at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:695)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:508)

Oddly, numpy is installed correctly. So that's the issue: numpy is installed correctly but pyspark isn't able to find it while creating a cosine-similarity matrix in a jupyte notebook.

Thank you for considering this.

Answer 1

If you are running Jupyter Notebook as an application within AWS EMR, try using a bootstrap script which installs the required version of numpy while provisioning the cluster