How to loop through multiple URLs in R and save in data frame

Question

I have been unable to loop through multiple URLs and have it save in a data frame. I have shared the code that can retrieve only one url at a time and save in a data frame.

The part of the url that changes is a number at the very end of the url that refers to a date. I am trying to scrape all the data from, for example, 20190901 through 20190915 and store it in the same data frame.

Here is the Code:

    library(rvest)
    library(dplyr)

    # Specifying URL
    url <- 'https://classic.sportsbookreview.com/betting-odds/mlb-baseball/?date=20190901'
    
    # Reading the HTML code from website
    oddspage <- read_html(url)

    # Using CSS selectors to scrape away teams
    awayHtml <- html_nodes(oddspage,'.eventLine-value:nth-child(1) a')

    #Using CSS selectors to scrape scores
    awayScoreHtml <- html_nodes(oddspage,'.first.total')
    awayScore <- html_text(awayScoreHtml)
    awayScore <- as.numeric(awayScore)
    homeScoreHtml <- html_nodes(oddspage, '.score-periods+ .score-periods .total')
    homeScore <- html_text(homeScoreHtml)
    homeScore <- as.numeric(homeScore)

    # Converting away data to text
    away <- html_text(awayHtml)

    # Using CSS selectors to scrape home teams
    homeHtml <- html_nodes(oddspage,'.eventLine-value+ .eventLine-value a')

    # Converting home data to text
    home <- html_text(homeHtml)

    # Using CSS selectors to scrape Away Odds
    awayPinnacleHtml <- html_nodes(oddspage,'.eventLine-consensus+ .eventLine-book.eventLine-book-value:nth-child(1) b')
    awayBookmakerHtml <- html_nodes(oddspage,'.eventLine-book:nth-child(12) .eventLine-book-value:nth-child(1) b')

    # Converting Away Odds to Text
    awayPinnacle <- html_text(awayPinnacleHtml)
    awayBookmaker <- html_text(awayBookmakerHtml)

    # Converting Away Odds to numeric
    awayPinnacle <- as.numeric(awayPinnacle)
    awayBookmaker <- as.numeric(awayBookmaker)

    # Using CSS selectors to scrape Pinnacle Home Odds
    homePinnacleHtml <- html_nodes(oddspage,'.eventLine-consensus+ .eventLine-book .eventLine-book-value+ .eventLine-book-value b')
    homeBookmakerHtml <- html_nodes(oddspage,'.eventLine-book:nth-child(12) .eventLine-book-value:nth-child(2) b')

    # Converting Home Odds to Text
    homePinnacle <- html_text(homePinnacleHtml)
    homeBookmaker <- html_text(homeBookmakerHtml)

    # Converting Home Odds to Numeric
    homePinnacle <- as.numeric(homePinnacle)
    homeBookmaker <- as.numeric(homeBookmaker)


    # Create Data Frame
    df <- data.frame(away,home,awayScore,homeScore,awayPinnacle,homePinnacle,awayBookmaker,homeBookmaker)

    View(df)

I am very new to coding and I have not been able to successfully apply any of the techniques used in similar questions.

Answer 1

Put all your code in a function and make date dynamic to generate url:

get_data <- function(date) {
      url <- paste0('https://classic.sportsbookreview.com/betting-odds/mlb-baseball/?date=', date)
      #...Rest of the code as it is
      #...
}

Create a vector of dates using sprintf

date_vec <- sprintf('201909%02d', 1:15)
date_vec
# [1] "20190901" "20190902" "20190903" "20190904" "20190905" "20190906"
# [7] "20190907" "20190908" "20190909" "20190910" "20190911" "20190912"
#[13] "20190913" "20190914" "20190915"

Use lapply to extract data for each date and combine them.

all_data <- do.call(rbind, lapply(date_vec, get_data))

You can also use map_df from purrr .

all_data <- purrr::map_df(date_vec, get_data)

However, you might need to add some checks in your function for pages which don't return any values for a particular field.