How to query hdfs from a spark cluster (2.1) which is running on kubernetes?
Question
I was trying to access HDFS files from a spark cluster which is running inside Kubernetes containers.
However I keep on getting the error: AnalysisException: 'The ORC data source must be used with Hive support enabled;'
What I am missing here?
1 answers
solution1
0 ACCPTED
Are you have SparkSession created with enableHiveSupport()?
Similar issue: Spark can access Hive table from pyspark but not from spark-submit
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Access hdfs cluster from pydoop
Submit an application to a standalone spark cluster running in GCP from Python notebook
Spark running in local machine instead of standalone cluster
Error with H20Context running PySparkling with Spark 2.1
How to avoid DiskPressure condition of pods and their eventual eviction while running Spark job on Kubernetes?
Managing Kubernetes cluster from GCP with python api
How to run a command in another pod in another namespace from an Airflow worker in a Kubernetes cluster
Create a non-hdfs csv from spark dataframe
Pass schema from hdfs file while creating Spark DataFrame
Minio in docker cluster is not reachable from spark container
Related Tags