如何使用Spark访问Hive数据

Question

I have table stored as text file eg employee in hive and I want to access it using spark.

1. First i have set sql context object using

    val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc) 

2. Then i have created table

    scala>sqlContext.sql("CREATE TABLE IF NOT EXISTS employee( id INT, name STRING, age INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\\n'") 

3. Further i was trying to load the contents of text file by using

    scala> sqlContext.sql("LOAD DATA LOCAL INPATH 'employee.txt' INTO TABLE employee") 

I am getting error as

SET hive.support.sql11.reserved.keywords=false
FAILED: SemanticException Line 1:23 Invalid path ''employee.txt'': No files 
matching path file:/home/username/employee.txt

If i have to place the textfile in current directory where the spark-shell is running how to do that ?

Answer 1

Do you run hive on hadoop? try to use absolute path... if this doesn't work, try to load your file to hdfs and then give absolute path to your file (hdfs location) .

Answer 2

Try doing the below steps

• Start spark-shell in local mode a Eg:spark-shell --master local[*]
• Give the file full path for loading file Eg:file:///home/username/employee.txt