将 JSON 从 S3 加载到 Redshift

Question

I have the following JSON data in an S3 bucket:

{
"campaigns": [
{"campaign_reach": 123456, 
"campaign_spend": 123456.0, 
"campaign_goal": 12345678, 
"id": "cda05a432b3b44c18c009a4a961f644a", 
"campaign_name": "Campaign1", 
"publisher_name": "PublisherA", 
"campaign_impressions": 123456}], 
"line_items": [], 
"podcasts": [
{"podcast_name": "PodcastA", "id": "86edbca2dc644ba8960c8f4bd55bdc19"}, 
{"podcast_name": "PodcastB", "id": "fc3f2dc4c20949edaaf2186613ec7e47"}]
}

I am using COPY to load the "campaigns" portion to a table in Redshift.

I have tried loading using jsonpaths

query_copy = """copy myschema.campaigns
from 's3://mybucket/mapping.json'
credentials 'aws_access_key_id=""" + acc + """;aws_secret_access_key=""" + sh + """'
json 's3://mybucket/campaign_jsonpaths.json'
;"""

My jsonpaths file "campaign_jsonpaths.json":

{
    "jsonpaths": [
        "$['id']",
        "$['campaign_name']",
        "$['campaign_reach'][0]",
        "$['campaign_spend']",
        "$['campaign_goal']",
        "$['campaign_impressions']",
        "$['publisher_name']",
    ]
}

I have also tried using json 'auto':

query_copy = """copy myschema.campaigns
from 's3://mybucket/mapping.json'
credentials 'aws_access_key_id=""" + acc + """;aws_secret_access_key=""" + sh + """'
json 'auto’
;"""

both result in successful runs, but the table in Redshift is empty. No errors in stl_load_errors.

I found a similar posting here, but no answers were provided: Redshift: copy command Json data from s3

Any help would be much appreciated.

Answer 1

I was able to load the table successfully by doing the following:

1. Created campaigns table based on your JSON data:

   create table campaigns ( id varchar(100), campaign_name varchar(100), campaign_reach int, campaign_spend float, campaign_goal int, campaign_impressions int, publisher_name varchar(100) );

2. Created a mapping.json file with your JSON data

3. Created a campaigns_jsonpaths.json as follows:

   { "jsonpaths": [ "$['campaigns'][0]['id']", "$['campaigns'][0]['campaign_name']", "$['campaigns'][0]['campaign_reach']", "$['campaigns'][0]['campaign_spend']", "$['campaigns'][0]['campaign_goal']", "$['campaigns'][0]['campaign_impressions']", "$['campaigns'][0]['publisher_name']" ] }

4. Ran copy:

   copy campaigns from 's3://<bucket>/mapping.json' iam_role 'arn:aws:iam::1234567890:role/Redshift-Role' json 's3://<bucket>/campaigns_jsonpaths.json';

Records were loaded successfully in the campaigns table.