将具有多种布局的json对象从S3复制到Redshift

Question

I have an S3 bucket with many files containing "\\n" delimited json objects. These json objects can have a few different layouts. There is a standard set of keys that are common across all the layouts. Most differences just have a few extra keys, but some have nested json objects. One file can have any/all of these layouts.

I have managed to define a single, basic table in Redshift and copy the data into that table, but any keys not in my table are lost.

I would like to create 1 table for each layout I have and have the json object copied into the appropriate table. The layouts with nested json objects could probably stay in a single string column as json since Redshift is able to parse json in a query.

I am new to AWS, so any help would be appreciated. Also, feel free to suggest non-Redshift services that might work as well.

Thanks!

Answer 1

You'll need to run a separate COPY for each table that you want to load. However you may have trouble with nested objects (as of right now).

We gave up on direct JSON loads because it cannot load an arbitrary number of nested objects. Each nested object has to be referred to by it's index (eg 'nest[0]' ) in order to load it. Which is not ideal when there could be many thousands of objects.

Answer 2

You cannot skip lines via the Copy command. One option you can think of is assuming that the files are being loaded to S3. You can split the file and land it into different folders. So that it you can run different copy commands to load the data to different tables. Other option: You can load just the first level json objects to a staging table and can use the Redshift JSON functions post that. Example: JSON 1:

{
    "a": "value",
    "b": "value",
    "c": "value",
    "d": "value",
    "f": {
        "fa": "value",
        "fb": "value",
        "fc": "value",
        "fd": "value"
    },
    "g": "value"
}

JSON 2:

{
    "a": "value",
    "b": "value",
    "c": "value",
    "d": "value",
    "e": {
        "ea": "value",
        "eb": "value",
        "ec": {
            "eca": "",
            "ecb": "value",
            "ecc": "value",
            "ecd": "value",
        }
    },
    "f": {
        "fa": "value"
    },
    "g": "value"
}

In JSON 2 , the 'e' tag is extra. You can load these two different types of JSONs to the same table and then use Redshift JSON functions to process further.

Your target table should have the columns:

a, b, c, d, e, f, g

Your jsonpath should look like:

{
    "jsonpaths": ["$.a", "$.b", "$.c", "$.d", "$.e", "$.f", "$.g"]
}

When JSON 1 is loaded, the column e is loaded as null.

I hope this is what you are looking for. Let me know if you found a solution.