R- using power.prop.test & prop.test

Question

Disclaimer: I have a similar thread open in Cross Validated, but it hasn't gotten any answers. I've decided to ask a simpler question here instead:

How can I use power.prop.test and prop.test together to determine an adequate sample size before an experiment and determine whether or not a conclusion is statistically significant afterwards?

Actually, any and all knowledge regarding these two functions (and related functions) would be very much appreciated.

Context: I'm trying to develop a testing methodology for simple A/B tests, ranging from experiment set-up to analysis.

Answer 1

You can write a function to draw a power curve to determine the sample size (per group) to achieve a desired power level, given your guess of population proportions.

Below is my crude attempt. This function gives you a data frame containing sample sizes and corresponding power levels, along with an optional plot. It takes the following arguments:

## n = vector of sample sizes;
## desired_power = the power level you want to achieve (typically 0.8);
## p1 = population proportion of group 1;
## p2 = population proportion of group 2;
## plot = whether you want a power curve or not (by default yes).

DrawPowerCurve <- function(n, desired_power, p1, p2, plot=TRUE){
  powers <- sapply(n, function(x) power.prop.test(x, p1=p1, p2=p2)$power)
  n_power <- min(n[powers>desired_power])
  print(data.frame(n, powers))

  if(plot){
    plot(n, powers, type="l")
    segments(y0=desired_power, x0=0, x1=n_power, col="red")
    segments(y0=0, y1=desired_power, x0=n_power, col="red")
    text(paste("n =", n_power, " \nper group"), x=n_power, y=desired_power/2, pos=4)
    title(paste("Sample Size (n) per Group to Achieve Power of", desired_power))
  }
}

Say you want to determine the sample size per group to achieve desired power of 0.8, given population proportions of 0.5 and 0.6. Then the plot shows that you'll need 390 participants per group.

n <- seq(10, 1000, 5)
DrawPowerCurve(n, desired_power=0.8, p1=0.5, p2=0.6, plot=TRUE)