Copy files (with specific suffix) from S3 to Redshift

Question

I am trying to import CSV files from S3 into a Redshift cluster. I have two types of files in the output folder in my bucket: 1. jobid_output.csv and jobid_maps.zip. I want to copy only output.csv files from S3 to RS.

This is my command:

copy mytable from 's3://mybucket/output/*output.csv' credentials 'aws_access_key_id=xxxxxxx;aws_secret_access_key=xxxxxxxx' region 'us-west-2' ignoreheader 1 fillrecord timeformat 'auto' dateformat 'auto' csv;

This is the error I'm getting: ERROR: XX000: The specified S3 prefix 'output/*output.csv' does not exist

I think I'm constructing the S3 URL wrong, but how should I do it?

Thanks

Answer 1

S3 does not support wildcards. The asterisk is a valid character in an S3 key name space. The asterisk is not a recommended character,but you can use it. You will have to implement directory searches at a higher level in your own code.