More efficient way to extract and subtract rows R in different dataframes

Question

I am working with this basketball game data with a dataframe games with around 50,000 rows. I am trying to compare the stats of each team ( A and B) in each game.

I have another dataframe called teamStats that has around 3000 rows each with teams from every season.

So far, I have assembled a code as the following:

    for (i in 1:nrow(games)) {
  if (length(which(((teamStats$Year == games$Season[i])==1) & (teamStats$teamID == games$teamA[i]))) == 1) {
    selectTeamA <- teamStats[which(((teamStats$Year == games$Season[i])==1) & (teamStats$teamID == games$teamA[i])),4:45]
  } else {
    selectTeamA <- as.numeric(rep(NA, ncol(differences)))
  }
  if (length(which(((teamStats$Year == games$Season[i])==1) & (teamStats$teamID == games$teamB[i]))) == 1) {
    selectTeamB <- teamStats[which(((teamStats$Year == games$Season[i])==1) & (teamStats$teamID == games$teamB[i])),4:45]
  } else {
    selectTeamB <- as.numeric(rep(NA, ncol(differences)))
  }

  differences[i,] <- selectTeamA - selectTeamB
}

Basically, this code searches for the correct teamID for each team A and B, after subsetting the correct season. Since every team for every season is not present in the teamstats, I have filled the missing rows with NA's for now. The "differences" dataframe is an empty dataframe that will be filled my the differences of stats of team A and B from the for loop.

To give you an idea of the data:

Games - first 6 rows

           Season teamA teamB winner scoreA scoreB
108123   2010  1143  1293      A     75     70
108124   2010  1198  1314      B     72     88
108125   2010  1108  1326      B     60    100
108126   2010  1107  1393      B     43     75
108127   2010  1143  1178      A     95     61

teamStats - first 6 rows and only first 6 columns for space - lot of columns with different stats in full dataframe. The code finds the correct row for the teamID and then substracts the stat columns such as GWL etc

              School Year teamID  G  W  L
1  abilene christian 2018   1101 32 16 16
2          air force 2018   1102 31 12 19
3              akron 2018   1103 32 14 18
4        alabama a&m 2018   1105 31  3 28
5 alabama-birmingham 2018   1412 33 20 13

And to close this very long post, my question. My for loop code works and fills the differences dataframe. The issue is it takes 20-30 minutes to run this code. I am not very experienced working with this much data. Is there a technique I don't know? How can I rewrite this code in a more efficient manner?

Answer 1

One approach is to merge games and teamStats , as an alternative to iterating across rows.

Some code to replicate your set-up, to create a minimal working example:

library(dplyr)
library(purrr)

set.seed(123)
n_games <- 50000
n_teams <- 400
n_years <- 10

games <- data.frame(Season = rep(2005:(2005 + n_years - 1),
                                 each = n_games / n_years)) %>%
  mutate(teamA = sample(1000:(1000 + n_teams - 1), n_games, r = TRUE),
         teamB = map_int(teamA, ~sample(setdiff(1000:(1000 + n_teams - 1), .), 1)),
         scoreA = as.integer(rnorm(n_games, 80, 20)),
         scoreB = as.integer(rnorm(n_games, 80, 20)),
         scoreB = ifelse(scoreA == scoreB, scoreA + sample(c(-1, 1), n_games, r = TRUE), scoreB),
         winner = ifelse(scoreA > scoreB, "A", "B"))

gen_random_string <- function(...) {
  paste(sample(c(letters, " "), rpois(1, 10), r = TRUE), collapse = "")
}

schools_ids <- data.frame(teamID = 1000:(1000 + n_teams - 1)) %>%
  mutate(School = map_chr(teamID, gen_random_string))

teamStats <- data.frame(Year = rep(2005:(2005 + n_years - 1), each = 300)) %>%
  mutate(teamID = as.vector(replicate(n_years, sample(schools_ids$teamID, 300))),
         G = 32, W = rpois(length(teamID), 16), L = G - W) %>%
  left_join(schools_ids)

We have games with 50k rows and teamStats with 3k rows. Now, we collapse teamStats into a tibble by Year and teamID :

teamStats <- teamStats %>%
  group_by(Year, teamID) %>%
  nest()

# # A tibble: 3,000 x 3
#     Year teamID data            
#    <int>  <int> <list>          
#  1  2005   1321 <tibble [1 x 4]>
#  2  2005   1192 <tibble [1 x 4]>
#  3  2005   1074 <tibble [1 x 4]>
# <snip>

Make a small convenience function to calculate differences:

calculate_diff <- function(x, y) {
  if (is.null(x) | is.null(y)) {
    data.frame(G = NA, W = NA, L = NA)
  } else {
    x[, 1:3] - y[, 1:3]
  }
}

Now, we (1) join (or merge) games with teamStats , (2) calculate differences using the joined dataset, and (3) unnest (or un-collapse) the dataframe.

start <- Sys.time()
differences <- games %>%
  left_join(teamStats, c("Season" = "Year", "teamA" = "teamID")) %>%
  rename(teamA_stats = data) %>%
  left_join(teamStats, c("Season" = "Year", "teamB" = "teamID")) %>%
  rename(teamB_stats = data) %>%
  mutate(diff = map2(teamA_stats, teamB_stats, calculate_diff)) %>%
  select(Season, teamA, teamB, diff) %>%
  unnest()
difftime(Sys.time(), start)
# Time difference of 11.27832 secs

with the result

head(differences)
#   Season teamA teamB  G  W  L
# 1   2005  1115  1085 NA NA NA
# 2   2005  1315  1177 NA NA NA
# 3   2005  1163  1051  0 -9  9
# 4   2005  1353  1190  0 -4  4
# 5   2005  1376  1286 NA NA NA
# 6   2005  1018  1362  0 -1  1

Answer 2

Here's an approach using tidyverse packages that I expect should be much faster than the loop solution in the OP. The speed (I expect) comes from relying more on database join operations (base merge or dplyr's left_join , for example) to connect the two tables.

library(tidyverse)

# First, use the first few columns from the `games` table, and convert to long format with
#   a row for each team, and a label column `team_cat` telling us if it's a teamA or teamB.
stat_differences <- games %>%
  select(row, Season, teamA, teamB)  %>% 
  gather(team_cat, teamID, teamA:teamB) %>%  

# Join to the teamStats table to bring in the team's total stats for that year
  left_join(teamStats %>% select(-row),    # We don't care about this "row"
            by = c("teamID", "Season" = "Year")) %>%

# Now I want to reverse the stats' sign if it's a teamB. To make this simpler, I gather
#   all the stats into long format so that we can do the reversal on all of them, and 
#   then spread back out.
  gather(stat, value, G:L) %>%
  mutate(value = if_else(team_cat == "teamB", value * -1, value * 1)) %>%
  spread(stat, value) %>%

# Get the difference in stats for each row in the original games table.
  group_by(row) %>%
  summarise_at(vars(G:W), sum)

# Finally, add the output to the original table
output <- games %>% 
  left_join(stat_differences)

To test this, I altered the given sample data so that the two tables would relate to each other:

games <- read.table(header = T, stringsAsFactors = F,
  text = "row           Season teamA teamB winner scoreA scoreB
108123   2010  1143  1293      A     75     70
108124   2010  1198  1314      B     72     88
108125   2010  1108  1326      B     60    100")

teamStats <- read.table(header = T, stringsAsFactors = F,
  text = "row   School Year teamID  G  W  L
1  abilene_christian 2010   1143 32 16 16
2          air_force 2010   1293 31 12 19
3              akron 2010   1314 32 14 18
4        alabama_a&m 2010   1198 31  3 28
5 alabama-birmingham 2010   1108 33 20 13
6       made_up_team 2018   1326 160 150 10    # To confirm getting right season
7       made_up_team 2010   1326 60 50 10"
)

Then I get the following output, which seems to make sense. (I just realized that the gather/mutate/spread I applied changed the order of the columns; if I have time I might try to use a mutate_if to preserve the order.)

> output
     row Season teamA teamB winner scoreA scoreB   G  L   W
1 108123   2010  1143  1293      A     75     70   1 -3   4
2 108124   2010  1198  1314      B     72     88  -1 10 -11
3 108125   2010  1108  1326      B     60    100 -27  3 -30