HBase使用Spark-SQL

Question

我在hbase中有一個名為“ sample”的表。 我需要使用Apache spark-sql查詢來查詢表。 有什么方法可以使用Apache Spark-sql查詢讀取hbase數據嗎？

Answer 1

Spark SQL是一個內存中查詢引擎，要在HBase表之上使用Spark SQL執行一些查詢操作，您需要

1. 使用Spark從HBase提取數據並創建Spark RDD

    SparkConf sparkConf = new SparkConf(); sparkConf.setAppName("SparkApp"); sparkConf.setMaster("local[*]"); JavaSparkContext javaSparkContext = new JavaSparkContext(sparkConf); Configuration config = HBaseConfiguration.create(); config.addResource(new Path("/etc/hbase/hbase-site.xml")); config.addResource(new Path("/etc/hadoop/core-site.xml")); config.set(TableInputFormat.INPUT_TABLE, "sample"); JavaPairRDD<ImmutableBytesWritable, Result> hbaseRDD = javaSparkContext.newAPIHadoopRDD(hbaseConfig, TableInputFormat.class, ImmutableBytesWritable.class, Result.class); JavaRDD<StudentBean> sampleRDD = hbaseRDD.map(new Function<Tuple2<ImmutableBytesWritable,Result>, StudentBean >() { private static final long serialVersionUID = -2021713021648730786L; public StudentBean call(Tuple2<ImmutableBytesWritable, Result> tuple) { StudentBean bean = new StudentBean (); Result result = tuple._2; bean.setRowKey(rowKey); bean.setFirstName(Bytes.toString(result.getValue(Bytes.toBytes("details"), Bytes.toBytes("firstName")))); bean.setLastName(Bytes.toString(result.getValue(Bytes.toBytes("details"), Bytes.toBytes("lastName")))); bean.setBranch(Bytes.toString(result.getValue(Bytes.toBytes("details"), Bytes.toBytes("branch")))); bean.setEmailId(Bytes.toString(result.getValue(Bytes.toBytes("details"), Bytes.toBytes("emailId")))); return bean; } }); 

2. 通過使用此RDD創建DataFrame對象，並使用一些臨時表名稱進行注冊，然后可以執行查詢

    DataFrame schema = sqlContext.createDataFrame(sampleRDD, StudentBean.class); schema.registerTempTable("spark_sql_temp_table"); DataFrame schemaRDD = sqlContext.sql("YOUR_QUERY_GOES_HERE"); JavaRDD<StudentBean> result = schemaRDD.toJavaRDD().map(new Function<Row, StudentBean>() { private static final long serialVersionUID = -2558736294883522519L; public StudentBean call(Row row) throws Exception { StudentBean bean = new StudentBean(); // Do the mapping stuff here return bean; } });