使用 aws.s3 包从 AWS S3 一次读取多个 CSV 文件对象

Question

I need to read multiple csv files from AWS S3 bucket with aws.s3 package in R and finally combine those files in single dataframe for further analysis.

Let' say I have several files in my S3 bucket like "variables_2019-08-12.csv" , "variables_2019-08-13.csv and "variables_2019-08-14.csv rtc.

I am using aws.s3::s3read_using but for object part I can only read one csv file each time. File has a date in their name so I was wondering how to add a loop in here :

my_file <- 
s3read_using(FUN = read_csv, object = "variables_2019-08-12.csv", bucket = "my_bucket")

Answer 1

There are many ways of doing this in R but the most intuitive for me is using map_dfr from the {purrr} package:

objects = c('variables_2019-08-12.csv', 'variables_2019-08-13.csv', …)
names(objects) = gsub('variables_(.*)\\.csv', '\\1', objects)
df = map_dfr(
    objects,
    ~ s3read_using(FUN = read_csv, object = .x, bucket = 'my_bucket'),
    .id = 'Date'
)

Because of the names(objects) assignment, and because we specify .id = 'Date' , the resulting data frame will have an additional column containing the date (based on the filenames) of each entry.