从 RStudio 将多个文件上传到 Google Cloud

Question

I am generating multiple html files using a combination of R and R markdown. Uploading multiple and individually identified files to AWS s3 is easy using a for loop and a put object command. Here's an example:

for (i in 1:nrow(Data)){
  rmarkdown::render(input = "report.Rmd",
                    output_format = "html_document",
                    output_file = paste("output/report_", i, ".html", sep =''))}
library(aws.s3)
Sys.setenv(AWS_ACCESS_KEY_ID = "XXXX",
           AWS_SECRET_ACCESS_KEY = "YYYYY",
           AWS_DEFAULT_REGION = "ZZZZ")

# put_object to upload to AWS
for (i in 1:nrow(Data)){
  put_object(file = paste("/Users/me/output/report_", i, ".html", sep = ''),
             object = paste("report_", i, ".html", sep = ''), bucket = "mybucket")}

I am trying to understand how to do the same with Google Cloud Storage.

library(googleCloudStorageR)
library(googleAuthR)


Sys.setenv("GCS_DEFAULT_BUCKET" = "reports",
           "GCS_AUTH_FILE" = "file/path/to.json")

I can upload a single output without issues:

gcs_upload(file = "output/report_1.html", bucket = "reports")

When I try to upload multiple, individually-named files using a loop it fails.

for (i in 1:nrow(Data)){
  gcs_upload(file = paste("output/report_", i, ".html", sep = ""), bucket = "reports")
}

This code will write the first html output, occasionally the second and never the complete set.

All help appreciated.

UPDATE: I have abandoned this effort. A workaround is to use a for loop to upload to AMZN and schedule a repeating transfer to GCS using GCS commands. Other workarounds use CLI but I don't want to go that way. I suspect the lack of response to this thread indicates there is little demand to create an upload from R to GCS. Separately, shout out to whomever wrote the R aws s3 package. Works beautifully!

Answer 1

I think it was probably due to not using a name argument in the loop, which by default derived from the name of the R object so when looping caused an error.

This would have probably worked:

for (i in 1:nrow(Data)){
  the_file_name <- paste("output/report_", i, ".html", sep = "")
  gcs_upload(the_file_name, name = the_file_name, bucket = "reports")
}