Access files on Amazon EC2 server with Python

Question

Apologies if this is a really simple question, but I cannot find any information on Google to help.

I have an Amazon EC2 server set-up, which collects data which is streamed to the server by some scientific instruments and saves this as.csv files.

I want to access these.csv files from a Python script running on my (remote) laptop. Is this possible?

My understanding is that I would need my Python code to 'login' to the server and then download the file to a local directory.

Any help/pointers would be gratefully recieved.

Update 1

I installed paramiko and tried the code below

import paramiko
paramiko.util.log_to_file("D:/Temp/aws/paramiko.log")

# Open a transport
host,port = "##.##.###.##",##
transport = paramiko.Transport((host,port))

# Auth    
username,password = "username","password"
transport.connect(None,username,password)

But, i get the following error:

SSHException: Error reading SSH protocol banner

The log file looks like this:

DEB [20210202-17:54:47.282] thr=1   paramiko.transport: starting thread (client mode): 0x5202388
DEB [20210202-17:54:47.283] thr=1   paramiko.transport: Local version/idstring: SSH-2.0-paramiko_2.7.2
DEB [20210202-17:54:47.384] thr=1   paramiko.transport: Banner: 220-FileZilla Server 0.9.60 beta
DEB [20210202-17:54:47.384] thr=1   paramiko.transport: Banner: 220-written by Tim Kosse (tim.kosse@filezilla-project.org)
DEB [20210202-17:54:47.384] thr=1   paramiko.transport: Banner: 220 Please visit https://filezilla-project.org/
DEB [20210202-17:54:48.003] thr=1   paramiko.transport: Banner: 500 Syntax error, command unrecognized.
ERR [20210202-17:54:50.019] thr=1   paramiko.transport: Exception: Error reading SSH protocol banner
ERR [20210202-17:54:50.026] thr=1   paramiko.transport: Traceback (most recent call last):
ERR [20210202-17:54:50.027] thr=1   paramiko.transport:   File "C:\Users\caira\Anaconda3\lib\site-packages\paramiko\transport.py", line 2211, in _check_banner
ERR [20210202-17:54:50.027] thr=1   paramiko.transport:     buf = self.packetizer.readline(timeout)
ERR [20210202-17:54:50.027] thr=1   paramiko.transport:   File "C:\Users\caira\Anaconda3\lib\site-packages\paramiko\packet.py", line 380, in readline
ERR [20210202-17:54:50.027] thr=1   paramiko.transport:     buf += self._read_timeout(timeout)
ERR [20210202-17:54:50.027] thr=1   paramiko.transport:   File "C:\Users\caira\Anaconda3\lib\site-packages\paramiko\packet.py", line 622, in _read_timeout
ERR [20210202-17:54:50.028] thr=1   paramiko.transport:     raise socket.timeout()
ERR [20210202-17:54:50.028] thr=1   paramiko.transport: socket.timeout
ERR [20210202-17:54:50.028] thr=1   paramiko.transport: 
ERR [20210202-17:54:50.028] thr=1   paramiko.transport: During handling of the above exception, another exception occurred:
ERR [20210202-17:54:50.028] thr=1   paramiko.transport: 
ERR [20210202-17:54:50.029] thr=1   paramiko.transport: Traceback (most recent call last):
ERR [20210202-17:54:50.029] thr=1   paramiko.transport:   File "C:\Users\caira\Anaconda3\lib\site-packages\paramiko\transport.py", line 2039, in run
ERR [20210202-17:54:50.029] thr=1   paramiko.transport:     self._check_banner()
ERR [20210202-17:54:50.029] thr=1   paramiko.transport:   File "C:\Users\caira\Anaconda3\lib\site-packages\paramiko\transport.py", line 2216, in _check_banner
ERR [20210202-17:54:50.029] thr=1   paramiko.transport:     "Error reading SSH protocol banner" + str(e)
ERR [20210202-17:54:50.029] thr=1   paramiko.transport: paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
ERR [20210202-17:54:50.029] thr=1   paramiko.transport: 

Am i doing something obvious that is wrong?

Should i be using the 'Public IPv4 DNS' instead? => ec2-##-##-###-##.us-east-2.compute.amazonaws.com

I am not sure if my username for above should be my full email address or my shorter username.

Answer 1

It is possible you can use SCP or SFTP to connect to the server and copy the data to your local machine. There are many ways to do that but for your use case you can use a library like Paramiko to handle the connection and the file transfer, if you like a sample code look at the answer https://stackoverflow.com/a/3635163/2156106

Update: For EC2 instance, you need to use the private key you downloaded or provided while you created the instance as follows

import paramiko

key = paramiko.RSAKey.from_private_key_file("path_to_key.pem")

transport = paramiko.Transport((host, port))
transport.connect(username="username", pkey=key)

sftp = paramiko.SFTPClient.from_transport(transport)
sftp.get(remote_path , local_path)

The username should be the username on the server. if it is an ubuntu server it is ubuntu by default in EC2.

Answer 2

you can use scp or sftp to do that ( https://www.ssh.com/ssh/sftp/ ), but I'd strongly recommend sending those files to aws s3 from the server (or even stream it).

you could then use awscli ( https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html ) or python's boto3 ( https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-download-file.html ) to copy the files from s3 to your local directory