将具有多个值的JSON从S3复制到Redshift

Question

I'm trying to load a JSON file with multiple values in one of the columns to Redshift using the copy command but get an error:

Invalid JSONPath format: Member is not an object.

This is how my JSON file looks like:

{"id":3,
"name":"John",
"children":[
{"child":"Ann","age":10},
{"child":"Dan","age":4},
{"child":"Ben","age":3}]
}    

This is my jsonpath file:

{
    "jsonpaths": [
        "$.id",
        "$.name",
        "$.children.child",
        "$.children.age"
    ]
}

And I expect the data in the SQL to show:

id      name     child    age     
--      ----     -----    ---

3       John     Ann      10 

3       John     Dan      4

3       John     Ben      3

Any ideas?

Answer 1

JSONPaths in COPY will only create one row per entry (or line) in your input file. In your example above, you want to create 3 lines from one entry, and Redshift doesn't support that.

You can consider preprocessing the data to convert it to something like this:

{
    "id":3,
    "name":"John",
    "child":"Ann",
    "age":10
},
{
    "id":3,
    "name":"John",
    "child":"Dan",
    "age":4
},
{
    "id":3,
    "name":"John",
    "child":"Ben",
    "age":3
}

And then, the following simple JSONPath will work:

{
    "jsonpaths": [
        "$.id",
        "$.name",
        "$.child",
        "$.age"
    ]
}

Answer 2

If you cannot change the structure of source file, try to load this to a PIVOT table with one column for each value (provided there are fixed number of values). Do an UNPIVOT in Redshift post the COPY process.