从R中的AWS S3存储桶获取最后修改的文件

Question

I've got a S3 bucket being updated in realtime with API data. The files are saved with a .XXX format, where xxx is 1...n.

My R script needs to be able to grab the latest files and add them to the analysis dataframe. I've been using the aws.s3 package so far. After setting secret/access keys to environment:

mybucket <- get_bucket("mybucket1")

Returns an s3 object of 1000 elements (presumably more), and it looks like each object has Contents:list if 7, one of which is $LastModified. How do I get the name of the last modified file?

Mybucket     Large s3_bucket (1000 elements, 2.1Mb)
contents:List of 7
..$ Key : chr "folder1"
..$ LastModified: chr "2018-01-16T09:58:47.000Z"
..$ ETag : chr "\" nnnnnnnnnnn\""
etc (.. $Owner, $Storage class, $bucket, $-attr)
contents: List of 7
..$ Key : chr "folder1/file.1
..$ LastModified: chr "2018....etc"
..$ ETag : chr "...etc..."
etc....
contents: List of 7
etc.....

It's really the number after 'file.' that I need (in this case it would be 1).

After experimentation, I think and CLI command through RCurl would be a better option.

aws s3 ls s3://mybucket --recursive | grep APIdata@symbol=XXX&interval=5.1*

This gets me really close, but the command is leaving out the '&interval=5.1*' so it's returning ALL objects with 'APIdata@symbol=XXX*'

Answer 1

I think your question is independent of AWS S3 and I would classify it as how can I create a data.frame from a list of lists and there are existing answers for that, eg:

R list of lists to data.frame

My solutions uses the handy rbindlist from the package data.table .

I had to guess about the data types of Mybucket but a solution could look like this:

# https://cran.r-project.org/web/packages/aws.s3/aws.s3.pdf
# get_bucket: returns a list of objects in the bucket (with class “s3_bucket”)
library(data.table)
library(lubridate)

# my personal assumption of the output of "get_bucket" is list of list (I have no S3 at hand to verify this)
Mybucket <- list(  list(Key = "folder1/file.1", LastModified = "2018-01-16T09:58:47.000Z", ETag = "\" nnnnnnnnnnn\"")
                 , list(Key = "folder2/file.2", LastModified = "2018-01-16T08:58:47.000Z", ETag = "xyz"))

dt <- rbindlist(Mybucket)  # convert into a data.table (enhanced data.frame)

dt[, LastModAsDate := ymd_hms(LastModified)]  # add a data column

dt.most.recent <- dt[order(-dt$LastModAsDate),][1]  # order by date descending, then pick the top-most row

which results in

> dt.most.recent
              Key             LastModified           ETag       LastModAsDate
1: folder1/file.1 2018-01-16T09:58:47.000Z " nnnnnnnnnnn" 2018-01-16 09:58:47

Please note that the date conversion may loose precision (milliseconds) but the overall solution is sketched anyhow...

To extract the number contained in the file extension use:

tools::file_ext(dt.most.recent$Key)
# [1] "1"

Answer 2

Easiest way ended up being with a system command:

currentfile <- system("aws s3 ls s3://bucket/folder --recursive | grep 'file.16' | sort | tail -n 1 | awk '{print $4}'", intern=TRUE)

grep grabs files with 'file.16' present, which significantly narrows the search as current file listings are in the 1600's. Intern=TRUE saves the response, in this case saves it in 'currentfile' as a character string. The sort, tail and print $4 orders files by modified date, takes last modified 4th column (name).

for reference: Downloading the latest file in an S3 bucket using AWS CLI?