如何在*服务器身份验证后使用RCurl *下载大型二进制文件

Question

i originally asked this question about performing this task with the httr package, but i don't think it's possible using httr . so i've re-written my code to use RCurl instead -- but i'm still tripping up on something probably related to the writefunction .. but i really don't understand why.

you should be able to reproduce my work by using the 32-bit version of R, so you hit memory limits if you read anything into RAM. i need a solution that downloads directly to the hard disk.

to start, this code to works -- the zipped file is appropriately saved to the disk.

library(RCurl)
filename <- tempfile()
f <- CFILE(filename, "wb")
url <- "http://www2.census.gov/acs2011_5yr/pums/csv_pus.zip"
curlPerform(url = url, writedata = f@ref)
close(f)
# 2.1 GB file successfully written to disk

now here's some RCurl code that does not work. as stated in the previous question , reproducing this exactly will require creating an extract on ipums .

your.email <- "email@address.com"
your.password <- "password"
extract.path <- "https://usa.ipums.org/usa-action/downloads/extract_files/some_file.csv.gz"

library(RCurl)

values <- 
    list(
        "login[email]" = your.email , 
        "login[password]" = your.password , 
        "login[is_for_login]" = 1
    )

curl = getCurlHandle()

curlSetOpt(
    cookiejar = 'cookies.txt', 
    followlocation = TRUE, 
    autoreferer = TRUE, 
    ssl.verifypeer = FALSE,
    curl = curl
)

params <- 
    list(
        "login[email]" = your.email , 
        "login[password]" = your.password , 
        "login[is_for_login]" = 1
    )

html <- postForm("https://usa.ipums.org/usa-action/users/validate_login", .params = params, curl = curl)
dl <- getURL( "https://usa.ipums.org/usa-action/extract_requests/download" , curl = curl)

and now that i'm logged in, try the same commands as above, but with the curl object to keep the cookies.

filename <- tempfile()
f <- CFILE(filename, mode = "wb")

this line breaks--

curlPerform(url = extract.path, writedata = f@ref, curl = curl)
close(f)

# the error is:
Error in curlPerform(url = extract.path, writedata = f@ref, curl = curl) : 
  embedded nul in string: [[binary jibberish here]]

the answer to my previous post referred me to this c-level writefunction answer, but i'm clueless about how to re-create that curl_writer C program (on windows?)..

dyn.load("curl_writer.so")
writer <- getNativeSymbolInfo("writer", PACKAGE="curl_writer")$address
curlPerform(URL=url, writefunction=writer)

..or why it's even necessary, given that the five lines of code at the top of this question work without anything crazy like getNativeSymbolInfo . i just don't understand why passing in that extra curl object that stores the authentication/cookies and tells it not to verify SSL would cause code that otherwise works.. to break?

Answer 1

1. From this link create a file named curl_writer.c and save it to C:\\<folder where you save your R files>

    #include <stdio.h> /** * Original code just sent some message to stderr */ size_t writer(void *buffer, size_t size, size_t nmemb, void *stream) { fwrite(buffer,size,nmemb,(FILE *)stream); return size * nmemb; } 

2. Open a command window, go to the folder where you saved curl_writer.c and run the R compiler

    c:> cd "C:\\<folder where you save your R files>" c:> R CMD SHLIB -o curl_writer.dll curl_writer.c 

3. Open R and run your script

    C:> R your.email <- "email@address.com" your.password <- "password" extract.path <- "https://usa.ipums.org/usa-action/downloads/extract_files/some_file.csv.gz" library(RCurl) values <- list( "login[email]" = your.email , "login[password]" = your.password , "login[is_for_login]" = 1 ) curl = getCurlHandle() curlSetOpt( cookiejar = 'cookies.txt', followlocation = TRUE, autoreferer = TRUE, ssl.verifypeer = FALSE, curl = curl ) params <- list( "login[email]" = your.email , "login[password]" = your.password , "login[is_for_login]" = 1 ) html <- postForm("https://usa.ipums.org/usa-action/users/validate_login", .params = params, curl = curl) dl <- getURL( "https://usa.ipums.org/usa-action/extract_requests/download" , curl = curl) # Load the DLL you created # "writer" is the name of the function # "curl_writer" is the name of the dll dyn.load("curl_writer.dll") writer <- getNativeSymbolInfo("writer", PACKAGE="curl_writer")$address # Note that "URL" parameter is upper case, in your code it is lowercase # I'm not sure if that has something to do # "writer" is the symbol defined above f <- CFILE(filename <- tempfile(), "wb") curlPerform(URL=url, writedata=f@ref, writefunction=writer, curl=curl) close(f) 

Answer 2

this is now possible with the httr package. thanks hadley!

https://github.com/hadley/httr/issues/44