没有截距的逻辑回归给出了合适的警告信息

Question

I am trying to run the logistic regression without an intercept. Firstly, I tried the function glm but I got the following error:

    Warning message:        
    glm.fit: fitted probabilities numerically 0 or 1 occurred       

Since it was not possible to change the data set at all given the nature of my work, I decided to use a different R program package which had the code bayesglm .

When I use this function including the intercept, I get no error message as above. However, when I exclude the intercept by adding -1 at the end of my function I still get the same error above with the following output:

    > regress=bayesglm(y~x1*x2+x3+x4-1, data = DATA, family=binomial(link="logit"))     
    > summary(regress)      

    Call:       
    bayesglm(formula = y ~ x1 * x2 + x3 + x4 - 1, family = binomial(link = "logit"),        
        data = DATA, maxit = 10000)     

    Deviance Residuals:         
         Min        1Q    Median        3Q       Max        
    -1.01451  -0.43143  -0.22778  -0.05431   2.89066        

    Coefficients:       
             Estimate Std. Error z value Pr(>|z|)           
    x1      -20.45537    9.70594  -2.108  0.03507 *         
    x2       -7.04844    2.87415  -2.452  0.01419 *         
    x1:x2     0.13409   17.57010   0.008  0.99391           
    x3       -0.17779    0.06377  -2.788  0.00531 **        
    x4       -0.02593    0.05313  -0.488  0.62548           
    ---     
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1      

    (Dispersion parameter for binomial family taken to be 1)        

        Null deviance: 494.91  on 357  degrees of freedom       
    Residual deviance: 124.93  on 352  degrees of freedom       
      (165 observations deleted due to missingness)     
    AIC: 134.93     

    Number of Fisher Scoring iterations: 123        

and get the same error as below:

    Warning message:        
    glm.fit: fitted probabilities numerically 0 or 1 occurred       

which I do not get if I do not add -1 to remove the intercept.

Therefore, I have two questions to ask:

1. Is it possible for me to ignore this warning message?

2. Otherwise, may I know how I can fix the problem according to this warning message?

Answer 1

The correct answer to this question is that the intercept should not be removed in a logistic regression. Fixing the warning message without fixing the mis-specification of the model is not appropriate practice.

In a logistic regression done properly, this error message can show up when there is perfect separation (combinations of predictors that completely explain class membership in the data sample at hand), and there are well established ways to deal with this phenomenon as explained for example on this page .

It is, however, inappropriate to remove the intercept in a logistic regression model. See this page and the extensive discussion in comments on the duplicate posting of this question on Cross Validated , in particular https://stats.stackexchange.com/questions/11109/how-to-deal-with-perfect-separation-in-logistic-regression which includes many suggestions.

Answer 2

I will try to provide an answer to the question.

What does the warning mean? The warning is given when numerical precision might be in question for certain observations. More precisely, it is given in the case where the fitted model, returns probability of 1 - epsilon or equivalently 0 + epsilon. As standard this bound is 1-10^-8 and 10^-8 respectively (as given by glm.control) for the standard glm.fit function.

When may this happen? To my experience the case where this happen most often, is the case where factors (or dummy variables) are included, for which only one outcome is observed in one catagory. This happens most often when interactions are included in factors of many levels, and limited data for the analysis. Similarly if one has many variables compared to the number of observations (counting used variables, interactions transformations etc. as individual variables, so the total number will be the sum of all of these), a similar image will be possible. In your case, if you have factors, removing the intercept will adds 1 level to each factor, which might reduce precision near the probability edge case of 0 and 1. In short if for some part of our data, we have no (or little) uncertainty, then this warning will give us an indication.

Can i ignore it otherwise how can i fix it? This is dependent on the problem at hand, and the scale of the problem. Several sources, like John Fox , will likely consider these observations possible outliers, and with good arguments suggests removing these after using influence measures (availible in the car package for base glm) or performing some outlier tests (also availible in the car package for base glm), if this is an option within your field of work. If these shows them to not influence the fit, you would not remove them, as there would be no statistical argument for doing so.

If outlier removal is not an option in your field of work, then a reduced model (less variables in general) might help if this is the cause, or if the number of factors are the cause merging levels within factors might give some better results.

Other sources might have other suggestions, but John Fox is a credible source on the subject for these model types. It becomes a question of 'Is my model correctly specified?', 'How severely does it affect my model?' and 'How much are you allowed to do in your line of work?', while following the general theory and guidelines within statistics. Probabilities close to 0 and 1 are less likely to be precise and more likely to be due to numerical impression, but if these are not the cases that you are likely to predict, and there is no significant effect on the remainder of the model, this is not necessarily a problem and may be ignored.