使用 curl 或 RCurl 在 R 中下载和解压缩 JSON 文件

Question

. Answers to this question are eligible for a +50 reputation bounty. Tom Wenseleers wants to to this question.

I have the following bash script to download & decompress a JSON file:

#!/bin/sh -ex

# Ensure data directory (or a link) exists.
test -e results || mkdir results

# Download and decompress data.
curl -u $GISAID_USERNAME:$GISAID_PASSWORD --retry 4 \
  https://www.epicov.org/epi3/3p/$GISAID_FEED/export/provision.json.xz \
  | xz -d -T8 > results/gisaid.json

Ideally I would like to have an R function to download & decompress this file in a given directory, with the environment variables above $GISAID_USERNAME, $GISAID_PASSWORD & $GISAID_FEED passed as arguments. Would anyone know how to accomplish this, eg using package curl or RCurl ? (It would also be OK not to decompress it and leave it as .json.xz, as I would be reading the file later using

library(jsonlite)
GISAID_json <- jsonlite::stream_in(gzfile(".//data//GISAID_json//provision.json.xz"))

Answer 1

I don't really know if this solves your problem, but what speaks against just executing your terminal commands in R using the system function?

So just put your terminal call into system() and it should execute and create your file. Afterwards read in the file. Of course you would have to replace the $GISAID_USERNAME, $GISAID_PASSWORD with your actual information. If the login information or the url should be flexible, you can put together a string beforehand, since system() expects a string with the command to execute.

system("curl -u $GISAID_USERNAME:$GISAID_PASSWORD --retry 4 \
https://www.epicov.org/epi3/3p/$GISAID_FEED/export/provision.json.xz \
| xz -d -T8 > results/gisaid.json")

Afterwards just read in the (hopefully) created file.

Couldn't test with your setup, but for me eg this small example successfully creates a file:

system("curl https://raw.githubusercontent.com/SteffenMoritz/imputeTS/master/pkgdown/favicon/favicon.ico > /Users/Steve/Downloads/x.ico")

Answer 2

Something like this should work:

library(curl)
library(glue)

custom_curl <- function(user, pass, feed, dest) {
  custom_handle <- curl::new_handle()
  curl::handle_setopt(
    custom_handle,
    username = user,
    password = pass
  )
  
  url <- glue::glue(
    "https://www.epicov.org/epi3/3p/{feed}/export/provision.json.xz"
  )
  
  curl::curl_download(url, dest, handle = custom_handle)
}

custom_curl('my_user', 'xxxxxx', 'feed1', 'dest/filename.json.xz')

As I can't test in the real files and url, I'm not sure if little tinkering in the function is needed, but at least is a starter point for you.