使用pyspark创建sparksession后是否需要停止spark？

Question

So I have this:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("NewSpark").getOrCreate()

I heard you need to stop spark once you're done but is this necessary in my case since it's just a python program?

Answer 1

The session will be closed if the spark object gets destroyed or if the script exits. So you shouldn't need to worry about "dangling connections" or anything like that.

However, if you have a bunch of non-spark work that you want to do at the end of the script, it may still be a good idea to stop the session early to avoid holding that connection open.

Note that you can use the SparkSession object as a context manager to automatically stop it at the end of a scope:

with SparkSession.builder.appName("NewSpark").getOrCreate() as spark:
    # do stuff
    # ...

# spark.stop() gets called automatically here