将应用程序提交到从Python笔记本在GCP中运行的独立Spark集群

Question

I am trying to submit a spark application to a standalone spark(2.1.1) cluster 3 VM running in GCP from my Python 3 notebook(running in local laptop) but for some reason spark session is throwing error "StandaloneAppClient$ClientEndpoint: Failed to connect to master sparkmaster:7077".

Environment Details: IPython and Spark Master are running in one GCP VM called "sparkmaster". 3 additional GCP VMs are running Spark Slaves and Cassandra Clusters. I connect from my local laptop(MBP) using Chrome to GCP VM IPython notebook in "sparkmaster"

Please note that terminal works:

bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.1.1 --master spark://sparkmaster:7077 ex.py 1000

Running it from Python Notebook:

import os
os.environ["PYSPARK_SUBMIT_ARGS"] = '--packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.1.1 pyspark-shell'

from pyspark.sql import SparkSession
from pyspark.sql.functions import *

spark=SparkSession.builder.master("spark://sparkmaster:7077").appName('somatic').getOrCreate() #This step works if make .master('local')

df = spark \
  .readStream \
  .format("kafka") \
  .option("kafka.bootstrap.servers", "kafka1:9092,kafka2:9092,kafka3:9092") \
  .option("subscribe", "gene") \
  .load()

so far I have tried these:

1. I have tried to change spark master node spark-defaults.conf and spark-env.sh to add SPARK_MASTER_IP.

2. Tried to find the STANDALONE_SPARK_MASTER_HOST=hostname -f setting so that I can remove "-f". For some reason my spark master ui shows FQDN:7077 not hostname:7077

3. passed FQDN as param to .master() and os.environ["PYSPARK_SUBMIT_ARGS"]

Please let me know if you need more details.

Answer 1

After doing some more research I was able to resolve the conflict. It was due to a simple environment variable called SPARK_HOME. In my case it was pointing to Conda's /bin(pyspark was running from this location) whereas my spark setup was present in a diff. path. The simple fix was to add export SPARK_HOME="/home/<>/spark/" to .bashrc file( I want this to be attached to my profile not to the spark session)

How I have done it:

Step 1: ssh to master node in my case it was same as ipython kernel/server VM in GCP

Step 2:

• cd ~
• sudo nano .bashrc
• scroll down to the last line and paste the below line
• export SPARK_HOME="/home/your/path/to/spark-2.1.1-bin-hadoop2.7/"
• ctrlX and Y and enter to save the changes

Note: I have also added few more details to the environment section for clarity.