How to use rclone to download data from S3

Question

I have a quick question on rclone.

I am trying to download data from tradestatistics.io , where it gives a sample code for downloading:

rclone sync spaces:tradestatistics/hs-rev1992-visualization hs-rev1992-visualization

My question is that how to access list of files in that source and can it be directly done with terminal?

Answer 1

Assuming you've already installed rclone ( https://rclone.org/downloads/ )

To configure rclone to see storage on S3, see https://rclone.org/s3/

Assuming spaces: is your correctly configured rclone source remote, you can list all files using terminal amnd rclone lsl command:

rclone lsl spaces:tradestatistics/hs-rev1992-visualization

where tradestatistics is the bucket and hs-rev1992-visualization is the root folder.

A more human-readable list can be done with lsf . It's not recursive, so add -R :

rclone lsf -R spaces:

More details at https://rclone.org/commands/rclone_lsl/ with info on other lists.

Answer 2

Somehow I could not figure out the configuration.
Let me show you the codes that I am using in my terminal (MacOS) to configure the rclone from the very beginning:

cd && curl -O https://downloads.rclone.org/rclone-current-osx-amd64.zip #this is for downloading latest version

unzip -a rclone-current-osx-amd64.zip && cd rclone-*-osx-amd64 #Unzipping

sudo mkdir -p /Users/USER/Desktop/test

sudo mv rclone /Users/USER/Desktop/test #Moving rclone to a new directory

rclone config #here is the start of misunderstanding

Then I did the following steps:

1. New remote
2. Name: spaces
3. Then I chose number 4 ( / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, SeaweedFS, and Tencent COS \ "s3")
4. Then again number 4 (4 / Digital Ocean Spaces\ "DigitalOcean")
5. Then it is asking for: Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars), where I am just choosing the default options.
6. then it is asking for AWS Access Key ID, where I left it blank
7. Then for AWS Secret Access Key (password), where I also left it blank
8. Endpoint for S3 API, where I am entering "tradestatistics.ams3.digitaloceanspaces.com"
9. Location constraint: I am leaving it blank as ""
10. then for security issues, I am choosing number 1, which is 1 / Owner gets FULL_CONTROL. No one else has access rights (default). \ "private"
11. For "Edit Advanced config" I am choosing "No, default"

And these are the all steps that I did for configuration and I can see this "spaces" in my current remotes. But when I enter codes like:

rclone lsl spaces:tradestatistics

rclone sync spaces:tradestatistics/hs-rev1992-visualization hs-rev1992-visualization 

It is returning the errors like:

2021/08/25 14:42:43 ERROR : S3 bucket tradestatistics path hs-rev1992-visualization: error reading source root directory: directory not found
2021/08/25 14:42:43 ERROR : Local file system at /Users/khumoyunusmanaliev/hs-rev1992-visualization: not deleting files as there were IO errors

Your assistance will be very helpful!