How to open spark web ui while running pyspark code in pycharm?

Question

I am running pyspark program in pycharm local on windows 10 machine. I want to open spark web ui to monitor job and understand metrics showed over spark web ui. While running same code on jupyter I can access web ui but when I run program using pycharm I do not see option to access spark web ui, even I tried url local:4040 or localhost:4041 but, this did not work.

Answer 1

You can configure which port that ui can access in sparksession .

Example:

spark = SparkSession.builder.master("local").\
appName("Word Count").\
config("spark.driver.bindAddress","localhost").\
config("spark.ui.port","4050").\
getOrCreate()

Now once the spark session initialized then we can access spark ui in http://localhost:4050/jobs/ until the pycharm job finished executing .

In case if the port is not able to access at the time then in your log spark prints where does we can access the spark ui .

Sample log:

2020-08-17 14:39:37 WARN Utils:66 - Service 'SparkUI' could not bind on port 4050. Attempting port 4051.

Answer 2

You have to maintain your spark session open.

To keep your session open within python script in Pycharm add input("Press enter to terminate") before your spark.stop() .

Until you press enter the session remains open & you can access spark Web UI.