I'm getting the error below when running spark-submit on a simple python script to import sparksession. I have the Spark_Home
user variable set to 'C:\\Spark\\spark-3.1.2-bin-hadoop3.2'; ENV path in user variables set to C:\\Spark\\spark-3.1.2-bin-hadoop3.2\\bin; HADOOP_HOME
path set to folder with WINUTILS.
Did everything here: https://phoenixnap.com/kb/install-spark-on-windows-10
Full code below.
Respectfully I tried to PATH issue: Could not find valid SPARK_HOME while searching this did not work, I'm on windows. This is work related, I'm new to stackoverflow and Spark. Thx!!
C:\Spark\spark-3.1.2-bin-hadoop3.2\bin>spark-submit C:\Spark\PYScripts\New.py
21/06/21 16:44:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Traceback (most recent call last):
File "C:/Spark/PYScripts/New.py", line 2, in <module>
from pyspark.sql import SparkSession
File "C:\Spark\spark-3.1.2-bin-hadoop3.2\python\lib\pyspark.zip\pyspark\__init__.py", line 53, in <module>
File "C:\Spark\spark-3.1.2-bin-hadoop3.2\python\lib\pyspark.zip\pyspark\rdd.py", line 34, in <module>
File "C:\Spark\spark-3.1.2-bin-hadoop3.2\python\lib\pyspark.zip\pyspark\java_gateway.py", line 31, in <module>
File "C:\Spark\spark-3.1.2-bin-hadoop3.2\python\lib\pyspark.zip\pyspark\find_spark_home.py", line 68
print("Could not find valid SPARK_HOME while searching {0}".format(paths), file=sys.stderr)
^
SyntaxError: invalid syntax
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Try to use the findspark
.
import findspark
findspark.find()
from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.master("local") \
.appName("Spark") \
.getOrCreate()
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.