将 JSON 数据的 CSV 文件从 S3 上传到 Redshift

Question

I have thousands of unusually formatted CSVs sitting in S3 that I need uploaded to Redshift.

The CSVs are formatted like so:

 Column A            Column B            ..... Column Z
{"id": 2034823"   "created": "2017-1-1"       "result": true} 

In other words, each row of the CSV is valid JSON.

I've tried a simple copy command, but to no avail. I tried to add the format as json 'auto'; flag, but still receiving errors:

Invalid Value: err_code 1216, line number 1, position 0

Is there a recommended way to handle CSVs in this format? I want to save them into an existing Redshift table that already has types defined

Answer 1

I have the same exact types of files. The steps I have followed to load them into a Redshift table like this

1. Create an external table in Redshift Spectrum table with struct
2. Insert into your Redshift table from the table above.

in your case

1. 
CREATE EXTERNAL TABLE <spectrum schema>.<your external table>
(
data struct<
id:integer,
created:timestamp,
...
result:varchar(5)>
)
row format serde 'org.openx.data.jsonserde.JsonSerDe'
with serdeproperties (
'dots.in.keys' = 'true',
'mapping.requesttime' = 'requesttimestamp')
 as location 's3:<your S3 bucket>';

2.
INSERT INTO <your Redshift table> 
SELECT data.id, data.created, ..., data.result
  FROM <your external table>

See how to setup Redshift Spectrum https://docs.aws.amazon.com/redshift/latest/dg/c-getting-started-using-spectrum.html

Let me know if you have further questions.