R：循环浏览链接列表

Question

I have some code that scrapes data off this link ( http://stats.ncaa.org/team/stats?org_id=575&sport_year_ctl_id=12280 ) and runs some calculations.

What I want to do is cycle through every team and collect and run the manipulations on every team. I have a dataframe with every team link, like the one above.

Psuedo code: for (link in teamlist) {scrape, manipulate, put into a table}

However, I can't figure out how to run loop through the links.

I've tried doing URL = teamlist$link[i], but I get an error when using readhtmltable(). I have no trouble manually pasting each team individual URL into the script, just only when trying to pull it from a table.

Current code:

library(XML)
library(gsubfn)

URL= 'http://stats.ncaa.org/team/stats?org_id=575&sport_year_ctl_id=12280'  
tx<- readLines(URL)
tx2<-gsub("</tbody>","",tx)
tx2<-gsub("<tfoot>","",tx2)
tx2<-gsub("</tfoot>","</tbody>",tx2)
Player_Stats = readHTMLTable(tx2,asText=TRUE, header = T, which = 2,stringsAsFactors = F)

Thanks.

Answer 1

I agree with @ialm that you should check out the rvest package, which makes it very fun and straightforward to loop through links. I will create some example code here using similar subject matter for you to check out.

Here I am generating a list of links that I will iterate through

rm(list=ls())
library(rvest)
mainweb="http://www.basketball-reference.com/"

urls=html("http://www.basketball-reference.com/teams") %>%
html_nodes("#active a") %>%
html_attrs()

Now that the list of links is complete I iterate through each link and pull a table from each

teamdata=c()
j=1
for(i in urls){
bball <- html(paste(mainweb, i, sep=""))
teamdata[j]= bball %>%
html_nodes(paste0("#", gsub("/teams/([A-Z]+)/$","\\1", urls[j], perl=TRUE))) %>%
html_table()
j=j+1
}

Answer 2

Please see the code below, which basically builds off your code and loops through two different team pages as identified by the vector team_codes . The tables are returned in a list where each list element corresponds to a team's table. However, the tables look like they will need more cleaning.

library(XML)
library(gsubfn)

Player_Stats <- list()
j <- 1
team_codes <-  c(575, 580)
for(code in team_codes) {

  URL <- paste0('http://stats.ncaa.org/team/stats?org_id=', code, '&sport_year_ctl_id=12280')
  tx<- readLines(URL)
  tx2<-gsub("</tbody>","",tx)
  tx2<-gsub("<tfoot>","",tx2)
  tx2<-gsub("</tfoot>","</tbody>",tx2)
  Player_Stats[[j]] = readHTMLTable(tx2,asText=TRUE, header = T, which = 2,stringsAsFactors = F)
  j <- j + 1

}