在R中下载csv文件

Question

I'm trying to download historical stock trading from my country with R. I tried with the download.file() function. Indeed, a file is downloaded but is an empty spreadsheet. Obviously, if I use this url in my browser the file I downloaded is in fact the one I want.

I would love to do it with quantmod, but that package only applies to larger markets

url<-"https://www.ccbolsa.cl/apps/script/detalleaccion/Transaccion.asp?Nemo=AFPCAPITAL&Menu=H"
destfile <- "/home/hector/TxHistoricas.xls"
download.file(url, destfile)

Thanks in advance.

Answer 1

You can jury-rig something like this if you don't want to use selenium:

library(rvest)
library(httr)
library(stringr)

URL <- "https://www.ccbolsa.cl/apps/script/detalleaccion/Transaccion.asp?Nemo=AFPCAPITAL&Menu=H"

Get initial URL:

res <- html_session(URL, timeout(30))

It embeds a form that it uses javascript to submit to get the form:

inputs <- html_nodes(res, "input")

It uses the last javascript entry to do a redirect on page load, so we need the location of it:

scripts <- html_nodes(res, "script")
action <- html_text(scripts[[length(scripts)]])

This is the new URL to submit to:

base_url <- "https://www.ccbolsa.cl/apps/script/detalleaccion"
loc <- str_match(action, '\\.action *= *"(.*)"')[,2]
doc_url <- sprintf("%s/%s", base_url, loc)

Gather up all the query params:

query <- lapply(inputs, xml_attr, "value")
names(query) <- sapply(inputs, xml_attr, "name")

Now we have to make a new POST request with the query encoded as "form", using and providing a redirect URL (timeout was necessary for me). This write the "xls" content to a file:

ret <- POST(doc_url, 
            body=query, 
            encode="form",
            add_headers(Referer=URL),
            write_disk("fil.xls", overwrite=TRUE),
            timeout(30))

It says it's an XLS file:

ret$headers$`content-type`
## [1] "application/vnd.ms-excel"

but it's really an HTML table, so you can really just do:

ret <- POST(doc_url, 
            body=query, 
            encode="form",
            add_headers(Referer=URL),
            timeout(30))

doc <- read_html(content(ret, as="text"))
dat <- html_table(html_nodes(doc, "table"), fill=TRUE)

to get what you're looking for (there are two ugly tables in the dat list and you may want to use header=TRUE as an additional parameter to html_table ).

I am not sure how "dynamic" this solution but that's test-able/verifiable.