SciPy error: RuntimeWarning: overflow encountered in _beta_ppf

Question

My SciPy version is 1.7.3 and I'm running on an Apple M1 chip (not sure if it's relevant). My Python version is 3.9.11, installed via Annaconda.

I get the error message /opt/anaconda3/lib/python3.9/site-packages/scipy/stats/_continuous_distns.py:624: RuntimeWarning: overflow encountered in _beta_ppf return _boost._beta_ppf(q, a, b) when running the following PyMC3 code:

import pymc3 as pm
from scipy import stats

Y = stats.bernoulli(0.7).rvs(20)

with pm.Model() as model:
    theta = pm.Beta("theta", alpha=0.1, beta=0.1)
    y_obs = pm.Binomial("y_obs", n=2, p=theta, observed=Y)
    idata = pm.sample(1000, return_inferencedata=True)

I don't think the issue is with PyMC3, because the PyMC3 code someone else wrote runs without any errors:

np.random.seed(123)
n_experiments=4
theta_real =0.35
data =stats.bernoulli.rvs(p=theta_real,size=n_experiments)
with pm.Model()  as our_first_model:
    theta =pm.Beta('theta',alpha=1,beta=1)
    y =pm.Bernoulli('y',p=theta,observed=data)
    start =pm.find_MAP()
    step =pm.Metropolis()
    trace =pm.sample(1000,step=step,start=start)
    burnin=100
    chain =trace[burnin:]
    pm.traceplot(chain,lines={'theta':theta_real})

The SciPy code snippet below, however, gives the same error:

import numpy as np
from scipy import stats
q = 0.999995
a = np.array([ 2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20])
b = np.array([99999, 99998, 99997, 99996, 99995, 99994, 99993, 99992, 99991, 99990, 99989, 99988, 99987, 99986, 99985, 99984, 99983, 99982, 99981])
stats.beta.ppf(q, a, b)

I Googled " overflow encountered in _beta_ppf " and found a GitHub issue ( https://github.com/scipy/scipy/issues/14901 ) that points to problems with SciPy <=1.7.2 on Intel chips. However, I'm using 1.7.3 on an M1 chip.

Anaconda doesn't allow me to update to the newest SciPy version (1.8.0). Using pip to update ( pip install scipy==1.8.0 ) results in incompatibilities with ArviZ ( arviz 0.11.2 requires typing-extensions<4,>=3.7.4.3, but you have typing-extensions 4.1.1 which is incompatible ). I guess I could downgrade typing-extensions , but it may cause further issues with other packages, so I'd rather not.

I wonder what I can do to solve this problem. Thanks!

Answer 1

If you cannot upgrade scipy to 1.8.0, you can always downgrade it to 1.6.x (IIRC boost is not used in 1.6.x series)

Answer 2

Native Conda Environment

FWIW, I use native Mambaforge for osx-arm64 , and have no problem with creating a native Python v3.10 with SciPy v1.8.0. If you already have an Anaconda/Miniconda base (ie, osx-64 ), you should be able to create a native environment with:

Shell Session

conda create -n arm64
conda activate arm64
conda config --env --add channel conda-forge
conda config --env --set subdir osx-arm64
conda install -y scipy=1.8 pymc3
python

Python Session

Python 3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:39:37) [Clang 12.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> from scipy import stats
>>> q = 0.999995
>>> a = np.array([ 2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20])
>>> b = np.array([99999, 99998, 99997, 99996, 99995, 99994, 99993, 99992, 99991, 99990, 99989, 99988, 99987, 99986, 99985, 99984, 99983, 99982, 99981])
>>> stats.beta.ppf(q, a, b)
array([0.00014976, 0.00017332, 0.00019478, 0.00021492, 0.0002341 ,
       0.00025256, 0.00027046, 0.00028788, 0.00030491, 0.00032161,
       0.00033802, 0.00035417, 0.00037009, 0.0003858 , 0.00040134,
       0.0004167 , 0.00043192, 0.00044699, 0.00046193])

Just make sure to activate the environment before installing additional packages.

---

Also, seems to not have any error with scipy=1.7.3 either.