NameError: name 'spark' is not defined，如何解决？

Question

I have just installed pyspark2.4.5 in my ubuntu18.04 laptop, and when I run following codes,

#this is a part of the code. 
import pubmed_parser as pp
from pyspark.sql import SparkSession
from pyspark.sql import Row

medline_files_rdd = spark.sparkContext.parallelize(glob('/mnt/hgfs/ShareDir/data/*.gz'), numSlices=1000)
parse_results_rdd = medline_files_rdd.\
    flatMap(lambda x: [Row(file_name=os.path.basename(x), **publication_dict)
                       for publication_dict in pp.parse_medline_xml(x)])

medline_df = parse_results_rdd.toDF()
# save to parquet
medline_df.write.parquet('raw_medline.parquet', mode='overwrite')


medline_df = spark.read.parquet('raw_medline.parquet')

I get such error,

medline_files_rdd = spark.sparkContext.parallelize(glob('/mnt/hgfs/ShareDir/data/*.gz'), numSlices=1000)
NameError: name 'spark' is not defined

I have seen similiar questions on StackOverflow, but all of them can not solve my problem.Does anyone can help me?Thanks a lot.

By the way, I am new in spark, if I just want to use spark in Python, does it enough that I just install pyspark by using pip install pyspark ? any others should I do? Should I install Hadoop or others?

Answer 1

Just create spark session in the starting

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('abc').getOrCreate()