R: Inflated degrees of freedom in mixed linear model

Question

This question was migrated from Stack Overflow because it can be answered on Cross Validated. Migrated 5 days ago .

I have a question regarding a mixed model I am using: In a study, participants have been presented with 40 different news article headlines and indicated for each headline whether they would share the headline or not (Yes coded as 1, No coded as 0). There are the two binary within-subjects factors “Accuracy” (true vs. false) and “Strategy” (attacks outgroup vs. praises ingroup). Further, there is a binary between-subjects factor “Condition” (threat vs. neutral).

I wanted to run a mixed model with random intercepts for participants (id) and headlines (Headline) that includes sharing decision as a dependent variable and Accuracy, Strategy and Condition as independent variables. I have two issues with that.

When I try to use a multilevel logistic regression with the following command, I am running into convergence issues:

mreg_P3_g <- glmer(
   Sharing_P3 ~ (1 | id) + (1 | Headline) + Strategy * Accuracy * Condition, 
   data=df,
   family="binomial"
)

Therefore, I tried to run a linear model with the following command:

mreg_P3 <- lmer(
  Sharing_P3 ~ Strategy * Accuracy * Condition + 
  (1|Headline) + (1|id),
  data=df
)

When I do that, I receive the following output:

        Type III Analysis of Variance Table with Kenward-Roger's method
                            Sum Sq Mean Sq NumDF  DenDF  F value    Pr(>F)    
Strategy                     0.828   0.828     1   35.1   7.5283  0.009505 ** 
Accuracy                    80.154  80.154     1   35.1 729.1441 < 2.2e-16 ***
Condition                    0.030   0.030     1  195.7   0.2728  0.602041    
Strategy:Accuracy            0.528   0.528     1   35.1   4.8006  0.035180 *  
Strategy:Condition           0.462   0.462     1 3723.1   4.2026  0.040431 *  
Accuracy:Condition           0.741   0.741     1 3723.1   6.7425  0.009451 ** 
Strategy:Accuracy:Condition  0.457   0.457     1 3723.1   4.1596  0.041468 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

As you probably see, I am having a lot of significant effects and the effects Strategy:Condition, Accuracy:Condition and Strategy:Accuracy:Condition are not interpretable when looking the visualized data. I attribute the fact that they are significant to the inflated degrees of freedom and wonder if I need to specify the random effects of my model differently.

I am far from an expert and would be very happy for any help! Thank you very much in advance!

Answer 1

I suspect that there is nothing wrong with the degrees-of-freedom (DF) estimates. If you have 100 participants each having 40 headlines to evaluate, you have 4000 observations. The DF used to evaluate those interaction terms should represent the number of observations minus the DF used up for other aspects of the model.

What seems more likely in your lmer() model is that you have "statistically significant" effects with your interaction terms that aren't practically significant, given the magnitude of the Accuracy effect. Practical and statistical significance aren't the same thing, particularly with large sample sizes.

That said, you should be paying attention to why the binomial model isn't converging. The lmer() model is seldom appropriate for binary outcomes and might give you probabilities below 0 or above 1. You don't say what the problem is, but logistic regression can run into perfect separation . It's also possible that the default solver or number of iterations weren't adequate for the size and nature of your data set. The above explanation of interaction effects that "are not interpretable when looking the visualized data" would still hold.