Kafka Stream to HDFS
public static void main(String[] args) throws Exception {
String brokers = "quickstart:9092";
String topics = "simple_topic_6";
String master = "local[*]";
SparkSession sparkSession = SparkSession
.builder().appName(EventKafkaToParquet.class.getName())
.master(master).getOrCreate();
SQLContext sqlContext = sparkSession.sqlContext();
SparkContext context = sparkSession.sparkContext();
context.setLogLevel("ERROR");
Dataset<Row> rawDataSet = sparkSession.readStream()
.format("kafka")
.option("kafka.bootstrap.servers", brokers)
.option("subscribe", topics).load();
rawDataSet.printSchema();
rawDataSet = rawDataSet.withColumn("employee", rawDataSet.col("value").cast(DataTypes.StringType));
rawDataSet.createOrReplaceTempView("basicView");
Dataset<Row> writeDataset = sqlContext.sql("select employee from basicView");
writeDataset
.repartition(1)
.writeStream()
.option("path","/user/cloudera/employee/")
.option("checkpointLocation", "/user/cloudera/employee.checkpoint/")
.format("parquet")
.trigger(Trigger.ProcessingTime(5000))
.start()
.awaitTermination();
}
External table on Hive
CREATE EXTERNAL TABLE employee_raw ( employee STRING )
STORED AS PARQUET
LOCATION '/user/cloudera/employee' ;
Now I want to create a HIVE view on top of employee_raw table, which gives the out put as
firstName, lastName, street, city, state, zip
The output of employee_raw table is
hive> select * from employee_raw;
OK
{"employee":{"firstName":"Ganesh","lastName":"Kumar","address":{"street":"1400 Dakota Dr","city":"Princeton","state":"NJ","zip":"09800"}}}
{"employee":{"firstName":"Ganesh","lastName":"Kumar","address":{"street":"1400 Dakota Dr","city":"Princeton","state":"NJ","zip":"09800"}}}
{"employee":{"firstName":"Ganesh","lastName":"Kumar","address":{"street":"1400 Dakota Dr","city":"Princeton","state":"NJ","zip":"09800"}}}
{"employee":{"firstName":"Ganesh","lastName":"Kumar","address":{"street":"1400 Dakota Dr","city":"Princeton","state":"NJ","zip":"09800"}}}
{"employee":{"firstName":"Ganesh","lastName":"Kumar","address":{"street":"1400 Dakota Dr","city":"Princeton","state":"NJ","zip":"09800"}}}
Time taken: 0.123 seconds, Fetched: 5 row(s)
Your inputs are appreciated
根据您的描述,我主要是想“ 从Hive中的JSON字符串中提取值 ”,因此您可以在链接的线程中找到答案。
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.