Chi-square test P-value from resampled method vs scipy.stats.chi2_contigency

Question

This question references to book "O'Relly Practical Statistics for Data Scientists 2nd Edition" chapter 3, session Chi-Square Test.

The book provides an example of one Chi-square test case, where it assumes a website with three different headlines that run by 1000 visitors. The result shows the # of clicks from each headline.

The Observed data is the following:

Headline   A    B    C
Click      14   8    12
No-click   986  992  988

The expected value is calculated in the following:

Headline   A        B        C
Click      11.13    11.13    11.13
No-click   988.67   988.67   988.67

The Pearson residual is defined as:

Where the table is now:

Headline   A        B        C
Click      0.792    -0.990   0.198
No-click   -0.085   0.106   -0.021

The Chi-square statistic is the sum of the squared Pearson residuals: . Which is 1.666

So far so good. Now here comes the resampling part:

1. Assuming a box of 34 ones and 2966 zeros
2. Shuffle, and take three samples of 1000 and count how many ones(Clicks)
3. Find the squared differences between the shuffled counts and expected counts then sum them.
4. Repeat steps 2 to 3, a few thousand times.
5. The P-value is how often does the resampled sum of squared deviations exceed the observed.

The resampling python test code is provided by the book as following: (Can be downloaded from https://github.com/gedeck/practical-statistics-for-data-scientists/tree/master/python/code )

## Practical Statistics for Data Scientists (Python)
## Chapter 3. Statistial Experiments and Significance Testing
# > (c) 2019 Peter C. Bruce, Andrew Bruce, Peter Gedeck

# Import required Python packages.

from pathlib import Path
import random

import pandas as pd
import numpy as np

from scipy import stats
import statsmodels.api as sm
import statsmodels.formula.api as smf
from statsmodels.stats import power

import matplotlib.pylab as plt

DATA = Path('.').resolve().parents[1] / 'data'

# Define paths to data sets. If you don't keep your data in the same directory as the code, adapt the path names.

CLICK_RATE_CSV = DATA / 'click_rates.csv'

...

## Chi-Square Test
### Chi-Square Test: A Resampling Approach

# Table 3-4
click_rate = pd.read_csv(CLICK_RATE_CSV)
clicks = click_rate.pivot(index='Click', columns='Headline', values='Rate')
print(clicks)

# Table 3-5
row_average = clicks.mean(axis=1)
pd.DataFrame({
    'Headline A': row_average,
    'Headline B': row_average,
    'Headline C': row_average,
})

# Resampling approach
box = [1] * 34
box.extend([0] * 2966)
random.shuffle(box)

def chi2(observed, expected):
    pearson_residuals = []
    for row, expect in zip(observed, expected):
        pearson_residuals.append([(observe - expect) ** 2 / expect
                                  for observe in row])
    # return sum of squares
    return np.sum(pearson_residuals)

expected_clicks = 34 / 3
expected_noclicks = 1000 - expected_clicks
expected = [34 / 3, 1000 - 34 / 3]
chi2observed = chi2(clicks.values, expected)

def perm_fun(box):
    sample_clicks = [sum(random.sample(box, 1000)),
                     sum(random.sample(box, 1000)),
                     sum(random.sample(box, 1000))]
    sample_noclicks = [1000 - n for n in sample_clicks]
    return chi2([sample_clicks, sample_noclicks], expected)

perm_chi2 = [perm_fun(box) for _ in range(2000)]

resampled_p_value = sum(perm_chi2 > chi2observed) / len(perm_chi2)

print(f'Observed chi2: {chi2observed:.4f}')
print(f'Resampled p-value: {resampled_p_value:.4f}')

chisq, pvalue, df, expected = stats.chi2_contingency(clicks)
print(f'Observed chi2: {chi2observed:.4f}')
print(f'p-value: {pvalue:.4f}')

Now, i ran the perm_fun(box) for 2,000 times and obtained a resampled P-value of 0.4775. However, if I ran perm_fun(box) for 10,000 times, and 100,000 times, I was able to obtain a resampled P-value of 0.84 both times. It seems to me the P-value should be around 0.84. Why is the stats.chi2_contigency showing a such smaller numbers?

The result I get for running 2000 times is:

Observed chi2: 1.6659
Resampled p-value: 0.8300
Observed chi2: 1.6659
p-value: 0.4348

And if I were to run it 10,000 times, the result is:

Observed chi2: 1.6659
Resampled p-value: 0.8386
Observed chi2: 1.6659
p-value: 0.4348

software version:

pandas.__version__:         0.25.1
numpy.__version__:          1.16.5
scipy.__version__:          1.3.1
statsmodels.__version__:    0.10.1
sys.version_info:           3.7.4

Answer 1

I ran your code trying 2000, 10000, and 100000 loops, and all three times I got close to .47. I did, however, get an error at this line that I had to fix:

resampled_p_value = sum(perm_chi2 > chi2observed) / len(perm_chi2)

Here perm_chi2 is a list and chi2observed is a float, so I wonder how this code ever ran for you (perhaps whatever you did to fix it was the source of the error). In any case, changing it to the intended

resampled_p_value = sum([1*(x > chi2observed) for x in perm_chi2]) / len(perm_chi2)

allowed me to run it and get close to .47.

Make sure that when you change the number of iterations, you do so only by changing the 2000, none of the other numbers.