How to copy Redshift S3 Unloaded Data on daily basis to Azure?

Question

I want to build a mechanism that will unload the data from redshift to S3 on daily basis in a folder and that data will get copied to Auzre on daily basis, I want to build it in such a way so that unload will take place on daily basis, and daily the old S3 file will get deleted so that Azure will take only the latest files in the Azure SQL Database. Can this be done. I am not able to find much on this.

Answer 1

ADF doesn't support s3 as sink. So you can't copy files to s3 with built-in copy activity. But when source is Redshift, ADF provides an unload setting which will help you to unload data to s3 first.

Then you could just focus on copying data from Redshift to azure sql. In you case, you could use schedule trigger or tumbling window trigger to build a daily based pipeline.