How to use hiveContext in streaming from executors?
Question
I'm trying to use an instance of HiveContext in a Spark streaming application (1.6), but it fails with the following exception:
java.lang.NullPointerException at org.apache.spark.sql.SQLConf.getConf(SQLConf.scala:638)
In particolar, I want to create a dataframe and execute a query for each processed message.
Someone can help me?
1 answers
solution1
0 2018-02-20 13:55:23
This error appears when you try to use SQLContext or HiveContext directly from executors.
(Spark 1.6)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
How to use Spark with HiveContext
How many executors should I use for spark streaming
How to get HiveContext from JavaSparkContext
In spark streaming job, how to collect error messages from executors to drivers and log these at the end of each streaming batch?
How jobs are assigned to executors in Spark Streaming?
How can I prune executors' logs in spark streaming
HiveContext not working from Oozie
How to increase the number of executors that process a certain stage in Spark Streaming/Kafka?
Getting the hivecontext from a dataframe
In Apache Spark SQL, How to close metastore connection from HiveContext
Related Tags