数据从 Azure blob 复制到 S3 和 Synapse 到 Redshift

Question

There is a requirement to copy from Azure Blob to S3 for 10TB data and also from Synpase to Redshift for 10TB of data.

What is the best way to achieve these 2 migrations?

Answer 1

For the Redshift - you could export Azure Synapse Analytics to aa blob storage in a compatible format ideally compressed and then copy the data to S3. It is pretty straightforward to import data from S3 to Redshift.

You may need a VM instance to load read from Azure Storage and put into AWS S3 (doesn't matter where). The simplest option seems to be using the default CLI (Azure and AWS) to read the content to the migration instance and write to to the target bucket. However me personally - I'd maybe create an application writing down checkpoints, if the migration process interrupts for any reason, the migration process wouldn't need to start from the scratch.

There are a few options you may "tweak" based on the files to move, if there are many small files or less large files, from which region to move where, ....

https://aws.amazon.com/premiumsupport/knowledge-center/s3-upload-large-files/

As well you may consider using the AWS S3 Transfer Acceleration , may or may not help too.

Please note every larger cloud provider has some outbound data egress cost , for 10TB it may be considerable cost