rclone - How do I list which directory has the latest files in AWS S3 bucket?

Question

I am currently using rclone accessing AWS S3 data, and since I don't use either one much I am not an expert.

I am accessing the public bucket unidata-nexrad-level2-chunks and there are 1000 folders I am looking at. To see these, I am using the windows command prompt and entering :

rclone lsf chunks:unidata-nexrad-level2-chunks/KEWX

Only one folder has realtime data being written to it at any time and that is the one I need to find. How do I determine which one is the one I need? I could run a check to see which folder has the newest data. But how can I do that?

The output from my command looks like this :

1/
10/
11/
12/
13/
14/
15/
16/
17/
18/
19/
2/
20/
21/
22/
23/
... ... ... (to 1000)

What can I do to find where the latest data is being written to? Since it is only one folder at a time, I hope it would be simple.

Edit : I realized I need a way to list the latest file (along with it's folder #) without listing every single file and timestamp possible in all 999 directories. I am starting a bounty and the correct answer that allows me to do this without slogging through all of them will be awarded the bounty. If it takes 20 minutes to list all contents from all 999 folders, it's useless as the next folder will be active by that time.

Answer 1

If you wanted to know the specific folder with the very latest file, you should write your own script that retrieves a list of ALL objects , then figures out which one is the latest and which bucket it is in. Here's a Python script that does it:

import boto3

s3_resource = boto3.resource('s3')

objects = s3_resource.Bucket('unidata-nexrad-level2-chunks').objects.filter(Prefix='KEWX/')

date_key_list = [(object.last_modified, object.key) for object in objects]

print(len(date_key_list)) # How many objects?

date_key_list.sort(reverse=True)

print(date_key_list[0][1])

Output:

43727
KEWX/125/20200912-071306-065-I

It takes a while to go through those 43,700 objects!