如何在 HIVE 中展平 json 结构？

Question

I have a JSON structure that looks like this (there is aa structure nested in a column):

HEADER:

{"user":{"location":"USA","id":1514008171,"name":"Auzzie Jet","screenname":"metalheadgrunge","geoenabled":false},"tweetmessage":"Anthrax - Black - Sunglasses hell","createddate":"2013-06-20T12:08:44","identifier":"1234","geolocation":null}

When I query this, this works:

SELECT *
FROM TBL a
WHERE header.identifier = '1234'

But then when I want to find the location (the nested json structure) it does not work:

SELECT *
FROM TBL a
WHERE header.identifier = '1234'
and a.header.user.location LIKE '%USA%'

Does anyone know how to query this in HIVE?

Answer 1

For flattening json structure you need to first create lateral view using json_tuple that's how you can be able to achieve what you intend to. Please find the complete solution how to do it.

Step 1 : Create external table tweets with single column tweet with data type of string.

CREATE EXTERNAL table tweets (tweet string);

Now put the json string {"user":{"location":"USA","id":1514008171,"name":"Auzzie Jet","screenname":"metalheadgrunge","geoenabled":false},"tweetmessage":"Anthrax - Black - Sunglasses hell","createddate":"2013-06-20T12:08:44","identifier":"1234","geolocation":null}

Step 2 : In text file named tweets.txt and run the below command to load the data from text file into hive table.

LOAD data local inpath 'tweets.txt' into table tweets;

Once done, now we are ready to play on our Json string.

So basically here what we are trying to achieve is querying on identifier and location fields which are basically at different levels.

user
    location    :   USA
    id          :   1514008171
    name        :   Auzzie Jet
    screenname  :   metalheadgrunge
    geoenabled  :   false
tweetmessage    :   Anthrax - Black - Sunglasses hell
createddate :   2013-06-20T12:08:44
identifier  :   1234
geolocation :   null

Level 1 fields are => user , tweetmessage , createddate , identifier , geolocation Level 2 Fields are => location , id , name , screenname , geoenabled

So firstly we need to create lateral View on Level 1 so that we can query on Level 1 fields. In our example we need to query on identifier . Also to query on Level 2 fields we need to explode our User view which would be possible by lateral view.

LATERAL VIEW json_tuple(t.tweet, 'user', 'identifier' ) t1 as `user`, `identifier`

and then to query on location , we need to create another lateral view for level 2 fields.

LATERAL VIEW json_tuple(t1.`user`,'name', 'location') t2 as `name`, `location`

and that's it finally we can use select on tweets with lateral views.

Step 3 and Final Query :

SELECT t.* FROM tweets t LATERAL VIEW json_tuple(t.tweet, 'user', 'identifier' ) t1 as `user`, `identifier` LATERAL VIEW json_tuple(t1.`user`,'name', 'location') t2 as `name`, `location` where t1.`identifier`=1234 and `location` ="USA";

For more read on lateral view : LateralView and on Json_Tuple : JsonTuple