A/B Test Duration & Sample Size Calculator Using pwr() Package in R

Question

I am using the googleAnalyticsR package in R to pull some website visit stats and calculate conversion rates. No problem thus far.

However, I have become stuck when trying to calculate the required population size and test duration using a package called pwr which I've modified based on some recommendations I found from another user online. Code below.

average_daily_traffic <-  10.63 #cvr$all_users/30
control <- 0.30721 #cvr$cvr_perc
uplift <- 0.01

sample_size_calculator <- function(control, uplift){
  variant <- (uplift + 1) * control
  baseline <- ES.h(control, variant)
  sample_size_output <- pwr.p.test(h = baseline,
                                   n = ,
                                   sig.level = 0.05,
                                   power = 0.8)
  if(variant >= 0)
  {return(sample_size_output)}
  else
  {paste("N/A")}
}


duration_calculator <- function(sample_size_output, average_daily_traffic){
  days_required <- c((sample_size_output)*2)/(average_daily_traffic)
  if(days_required >= 0)
  {paste0("It will take approximately ", round(days_required, digits = 0), " days or ", round(round(days_required, digits = 0)/365, digits = 0) ," years for this test to reach significance, based on average traffic in the last 30 days")}
  else
  {paste("N/A")}
}


sample_size_calculator <- sample_size_calculator(control, uplift)
sample_size_output <-   sample_size_calculator$n
sample_size_output

duration_calculator(sample_size_output, average_daily_traffic)

The recommendation I saw online was to create 2 functions. One called 'sample_size_calculator' and the other called 'days_calculator', with both being fairly self-explanatory. It is at least clear to me what the intended function of both is.

My output is thus:

[1] "It will take approximately 33394 days or 91 years for this test to reach significance, based on average traffic in the last 30 days"

This seemed fairly realistic to me until I tried to verify my results using a couple of other online tools including VWO , Unbounce and AB Tasty , all of which suggest that I'm ~0.5x away from where I should be in terms of the number of days required to run the test. I appreciate some of the variance between the aforementioned calculators will be due to how each of the formulae handle rounding, but I'm more concerned with why and where I've gone wrong in my calculations such as to underestimate the test duration by half.

I could simply multiply the resultant number by two and go to bed but I'm keen to understand my error or even learn a more statistically and syntactically elegant way of coding this.

Thanks in advance.

Answer 1

Personally, I'd suggest to try to simulate data instead of relying on pre-packaged power calculations. You seem to have a good grasp of writing functions in R, so it won't be much of a step-up for you to simulate data using iteration (eg with for loops, or, what I'd personally recommend more, vectorized iteration with purrr ).

The advantage of simulating data is that it forces you to think your model through in advance, which can be invaluable later on when you're modelling the real data.

This is a great, if a little bit dated, tutorial: http://disjointedthinking.jeffhughes.ca/2017/09/power-simulations-r/