使用下拉列表抓取历史数据

Question

I try to scrape data from https://www.accessdata.fda.gov/scripts/cder/daf/index.cfm?event=reportsSearch.process . As you see, there is a drop-down menu to call historical data. However, the link is not associated with the date range. Hence, I am not able to create a rvest loop going to relevant date and scraping data. How can I get the historical drug approval data from this page under these circumstances?

Answer 1

The url, eg, for July 2019 appears to be " https://www.accessdata.fda.gov/scripts/cder/daf/index.cfm?event=reportsSearch.process&rptName=0& reportSelectMonth =7& reportSelectYear =2019" so you could create a loop for months and years, apply them to the reportSelectMonth and reportSelectYear portions of the url, and invoke read_html() on each of the dynamically created url?

If you want all of 2017 and 2018, for example, you could do:

library(rvest)
baseUrl <- "https://www.accessdata.fda.gov/scripts/cder/daf/index.cfm?event=reportsSearch.process&rptName=0&"
for (year in 2017:2018) {
    for (month in 1:12) {
        url <- paste0(baseUrl, "&reportSelectMonth=", month, "&reportSelectYear=", year)
        p <- read_html(url)
        # do stuff
    } 
}