AWS | 从 SFTP 拉取数据

Question

I'm planning to write a data pipeline that pull the data from on-prem SFTP server to S3. How can I achieve this? I am interested in AWS specific services only.

Answer 1

AWS provides a native way to transfer your files from any on-prem servers to s3 directly AWS Transfer Family . It is a multi-AZ, highly available, massively scaling SFTP interface for S3. With this basically you will running a cron or scheduler process to transfer the on prem SFTP server files into S3 buckets via AWS SFTP server.

Documentation link - AWS Transfer Family

The only catch here is the pricing for it. It is little higher on the cost. There are third party tools available with lesser price like SFTP Gateway for AWS which can be found in AWS marketplace

Documentation link - SFTP Gateway

Lastly, if your use case is simple with only very few SFTP uploads a day, solution would be to use T2/T3 EC2 instances and a cronjob to run the custom developed pipeline script that pulls data from on prem SFTP server.

Answer 2

Assuming that you have a key file to connect to your AWS instance, you can use scp to specify it in the file transfer command:

scp -i ~/.ssh/<aws_key>.pem <path_to_source_file> <aws_user>@<aws_ip>:<destination_path>

Answer 3

I'm planning to write a data pipeline that pull the data from on-prem SFTP server to S3. How can I achieve this? I am interested in AWS specific services only.