R bootstrap weighted mean by group with data table

Question

I am trying to combine two approaches:

1. Bootstrapping multiple columns in data.table in a scalable fashion

with

2. Bootstrap weighted mean in R

Here is some random data:

## Generate sample data

# Function to randomly generate weights
set.seed(7)
rtnorm <- function(n, mean, sd, a = -Inf, b = Inf){
qnorm(runif(n, pnorm(a, mean, sd), pnorm(b, mean, sd)), mean, sd)
}

# Generate variables
nps    <- round(runif(3500, min=-1, max=1), 0) # nps value which takes 1, 0 or -1
group  <- sample(letters[1:11], 3500, TRUE) # groups
weight <- rtnorm(n=3500, mean=1, sd=1, a=0.04, b=16) # weights between 0.04 and 16

# Build data frame
df = data.frame(group, nps, weight)

# The following packages / libraries are required:
require("data.table")
require("boot")

This is the code from the first post above boostrapping the weighted mean:

samplewmean <- function(d, i, j) {
  d <- d[i, ]
  w <- j[i, ]
  return(weighted.mean(d, w))   
}

results_qsec <- boot(data= df[, 2, drop = FALSE], 
                     statistic = samplewmean, 
                     R=10000, 
                     j = df[, 3 , drop = FALSE])

This works totally fine.

Below ist the code from the second post above bootstrapping the mean by groups within a data table:

dt = data.table(df)
stat <- function(x, i) {x[i, (m=mean(nps))]}
dt[, list(list(boot(.SD, stat, R = 100))), by = group]$V1

This, too, works fine.

I have trouble combining both approaches:

Running …

dt[, list(list(boot(.SD, samplewmean, R = 5000, j = dt[, 3 , drop = FALSE]))), by = group]$V1

… brings up the error message:

Error in weighted.mean.default(d, w) : 
  'x' and 'w' must have the same length

Running …

dt[, list(list(boot(dt[, 2 , drop = FALSE], samplewmean, R = 5000, j = dt[, 3 , drop = FALSE]))), by = group]$V1

… brings up a different error:

Error in weighted.mean.default(d, w) : 
  (list) object cannot be coerced to type 'double'

I still have problems getting my head around the arguments in data.table and how to combine functions running data.table.

I would appreciate any help.

Answer 1

It is related to how data.table behaves within the scope of a function. d is still a data.table within samplewmean even after subsetting with i whereas weighted.mean is expecting numerical vector of weights and of values. If you unlist before calling weighted.mean , you will be able to fix this error

Error in weighted.mean.default(d, w) : (list) object cannot be coerced to type 'double'

Code to unlist before passing into weighted.mean :

samplewmean <- function(d, i, j) {
  d <- d[i, ]
  w <- j[i, ]
  return(weighted.mean(unlist(d), unlist(w)))   
}

dt[, list(list(boot(dt[, 2 , drop = FALSE], samplewmean, R = 5000, j = dt[, 3 , drop = FALSE]))), by = group]$V1

A more data.table -like (data.table version >= v1.10.2) syntax is probably as follows:

#a variable named original is being passed in from somewhere and i am unable to figure out from where
samplewmean <- function(d, valCol, wgtCol, original) {
    weighted.mean(unlist(d[, ..valCol]), unlist(d[, ..wgtCol]))
}

dt[, list(list(boot(.SD, statistic=samplewmean, R=1, valCol="nps", wgtCol="weight"))), by=group]$V1

Or another possible syntax is: (see data.table faq 1.6 )

samplewmean <- function(d, valCol, wgtCol, original) {
    weighted.mean(unlist(d[, eval(substitute(valCol))]), unlist(d[, eval(substitute(wgtCol))]))
}

dt[, list(list(boot(.SD, statistic=samplewmean, R=1, valCol=nps, wgtCol=weight))), by=group]$V1