Post hoc test in R for repeated measures ANOVA with 2 within-variables

Question

Say, I have a full factorial design with 30 participants (ppt_id), each of which was tested in 4 conditions:

(left side + blue color) AND (left side + red color) AND (right side + blue color) AND (right side + red color)

And my dependent variable is reaction time (RT).

df = matrix(nrow = 120, ncol = 0) 
df = as.data.frame(df)
df$ppt_id = c(rep(c(1:30),4))
df$color = c(rep("red", 60), rep("blue", 60))
df$side = c(rep(c(rep("left", 30), rep("right", 30)),2))
df$RT = c(rnorm(30, 600, 50),
          rnorm(30, 650, 50),
          rnorm(30, 700, 50),
          rnorm(30, 600, 50))

Now, I calculate rmANOVA and find out that there is a significant interaction between factors Side and Color:

anova_test(
  data = df, 
  dv = RT, 
  wid = ppt_id,
  within = c(color, side),
  effect.size = "pes")

How can I perform multiple comparisons of all conditions with each other (a post hoc test) in R? All I found were functions for one variable with more than two levels (one-way ANOVA) or for mixed designs (ie, one between- and one within-variable).

Answer 1

One option may be to use the afex package to fit the ANOVA and the marginaleffects package to compute contrasts. (Disclaimer: I am the author of marginaleffects .)

First, fit the model:

library(marginaleffects)
library(afex)

df = matrix(nrow = 120, ncol = 0) 
df = as.data.frame(df)
df$ppt_id = c(rep(c(1:30),4))
df$color = c(rep("red", 60), rep("blue", 60))
df$side = c(rep(c(rep("left", 30), rep("right", 30)),2))
df$RT = c(rnorm(30, 600, 50),
          rnorm(30, 650, 50),
          rnorm(30, 700, 50),
          rnorm(30, 600, 50))

mod <- aov_ez(
    id = "ppt_id",
    dv = "RT",
    within = c("side", "color"),
    data = df)

Then, do post hoc processing to get the contrasts. This will tell you how the values of the outcome predicted by the model change when we manipulate the explanators (and their pairwise combinations):

cmp <- comparisons(
    # the model
    mod, 
    # hold other regressors at their means if there are any
    newdata = "mean",
    # vector of variables we want to "manipulate" in the contrast
    variables = c("side", "color"),
    # we care about interactions
    interactions = TRUE)

summary(cmp)
#> Average contrasts 
#>           side      color  Effect Std. Error z value   Pr(>|z|)
#> 1  left - left blue - red 110.489      10.71 10.3148 < 2.22e-16
#> 2 right - left  red - red  51.055      13.84  3.6896 0.00022457
#> 3 right - left blue - red   6.274      12.19  0.5147 0.60674367
#>    2.5 % 97.5 %
#> 1  89.49 131.48
#> 2  23.93  78.18
#> 3 -17.62  30.16
#> 
#> Model type:  afex_aov 
#> Prediction type:  response

There is a detailed vignette on contrasts here: https://vincentarelbundock.github.io/marginaleffects/articles/contrasts.html

Answer 2

Perhaps I'm not able to understand the answer by @Vincent, or maybe I did not explain clearly what I needed.

Ideally, I would like to get a table with all possible comparisons for this dataset, ie, there should be six comparisons (because there are 4 possible conditions), something like what this function does:

## combine both independent variables into one
df$condition = paste0(df$side, "_", df$color)
df$condition = as.factor(df$condition)

## calculate pairwise tests
mult_comp = df %>%
  pairwise_t_test(
    RT ~ condition, paired = TRUE, 
    p.adjust.method = "holm")%>%
  select(-statistic, -df)

Now, the problem is that this function can only work with one independent variable (the same is true for a standard function pairwise.t.test), while I actually have two independent variables.

One possible solution is to calculate ANOVA by using the function aov and then use the function TukeyHSD for calculating pairwise comparisons:

anova_df = aov(RT ~ side*color, data = df)

TukeyHSD(anova_df)

The downside is that the calculation is then limited to the Tukey method, which might not always be appropriate. It is not possible to use other corrections with this function.