Hive/Athena 中的 JSON SerDe：将一个 JSON object 变成多行？

Question

I am looking into using AWS Athena to do queries against a mass of JSON files.

My JSON files have this format (prettyprinted for convenience):

{
 "data":[
         {<ROW1>},
         {<ROW2>},
          ...
        ],
 "foo":[...],
 "bar":[...]
}

The ROWs contained in the "data" array are what should be queried. The rest of the JSON file is unimportant.

Can this be done without modifying the JSON files? If yes, how? From what I've been able to find, looks like the SerDes (or is it Hive itself?) assume one row of output per line of input, which would mean that I'm stuck with modifying all my JSON files (and turning them into JSONL?) before uploading them to S3.

(Athena uses the Hive JSON SerDe and the OpenX JSON SerDe; AFAICT, there is no option to write my own SerDe or file format...)

Answer 1

You can't make the serde do it automatically, but you can achieve what you're after in a query. You can then create a view to simulate a table with the data elements unwrapped.

The way you do this is to use the UNNEST keyword. This produces one new row per element in an array:

SELECT
  foo,
  bar,
  element
FROM my_table, UNNEST(data) AS t(element)

If your JSON looked like this:

{"foo": "f1", "bar": "b1", "data": [1, 2, 3]}
{"foo": "f2", "bar": "b2", "data": [4, 5]}

The result of the query would look like this:

foo | bar | element
----+-----+--------
f1  | b1  | 1
f1  | b1  | 2
f1  | b1  | 3
f2  | b2  | 4
f2  | b2  | 5