I have the following code. What I am trying to achieve is to check if a table already exists in a database in Hive. If yes, then append new data into. If not then create a new table.
if "table1" in sqlContext.tableNames("db1"):
df.write.format("orc").mode("append").insertInto("db1.table1")
else:
df.write.format("orc").saveAsTable("db1.table1")
But I am getting this error:
NameError: name 'sqlContext' is not defined
I am using spark session in Pyspark with Jupyter Notebook:
spark = SparkSession.builder.config(conf=conf).enableHiveSupport().getOrCreate()
How can I fix the error?
It depends on your Apache Spark version. If your Spark version is 1.x then you need to do as follows:
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.