[英]Apache Spark - ModuleNotFoundError: No module named 'mysql'
I'm trying to submit Apache Spark driver program to the remote cluster.我正在尝试将 Apache Spark 驱动程序提交到远程集群。 I'm having difficulties with the python package called
mysql
.我在使用名为
mysql
的 python package 时遇到了困难。 I installed this package on all Spark nodes.我在所有 Spark 节点上安装了这个 package。 Cluster is running inside docker-compose, images are based on bde2020 .
集群在 docker-compose 内部运行,图像基于bde2020 。
$ docker-compose logs impressions-agg
impressions-agg_1 | Submit application /app/app.py to Spark master spark://spark-master:7077
impressions-agg_1 | Passing arguments
impressions-agg_1 | 19/11/13 18:45:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
impressions-agg_1 | Traceback (most recent call last):
impressions-agg_1 | File "/app/app.py", line 6, in <module>
impressions-agg_1 | from mysql.connector import connect
impressions-agg_1 | ModuleNotFoundError: No module named 'mysql'
impressions-agg_1 | log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
impressions-agg_1 | log4j:WARN Please initialize the log4j system properly.
impressions-agg_1 | log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Module mysql
is installed via pip on all nodes.模块
mysql
通过 pip 安装在所有节点上。
$ docker-compose exec spark-master pip list
Package Version
--------------- -------------------
mysql-connector 2.2.9
pip 18.1
setuptools 40.8.0.post20190503
$ docker-compose exec spark-worker pip list
Package Version
--------------- -------------------
mysql-connector 2.2.9
pip 18.1
setuptools 40.8.0.post20190503
How can I solve this?我该如何解决这个问题? Thank you for any information.
感谢您提供任何信息。
While the node has mysql installed, the container does not.虽然节点安装了 mysql,但容器没有。 What the logs are telling you is that
impressions-agg_1
contains a script at /app/app.py
which is trying to load mysql but cannot find it.日志告诉您的是,
impressions-agg_1
在/app/app.py
中包含一个脚本,该脚本试图加载 mysql 但找不到它。
Did you create impressions-agg_1
?您是否创建了
impressions-agg_1
? Add a RUN pip install mysql
step to its Dockerfile.将
RUN pip install mysql
步骤添加到其 Dockerfile。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.