Function Maximisation

Question

I'm trying to maximise a function using the following code:

from scipy.optimize import minimize
from scipy.stats import lognorm, norm
import numpy as np

np.random.seed(123)
obs = np.random.normal(loc=20, scale=3, size=20)

# Log-Posterior optimisation objectiv 
def objective(params, y):
    mu = params[0]
    sigma = params[1]
    llikelihood = np.sum(np.log(norm.pdf(y, mu, sigma)))
    lpost = llikelihood + np.log(norm.pdf(mu, 0, 100)) + np.log(lognorm.pdf(sigma, loc= 0, s = 4))
    return -1*lpost

starting_mu = 0
starting_sigma = 1
optim_res = minimize(fun = objective, x0=(starting_mu, starting_sigma), args=(obs))

The code runs fine up to the final optimisation line. I'm quite confident that the error is how I am trying to do the optimisation as in R, using identical setup and observations, objective() evaluates to the same value. Additionally, using optim() the function does optimise to value of mu=21.6 and sigma=3.28.

I can use the R code, however, it would be easier to run the code in Python so that it could integrate with everything else that I am doing.

EDIT: The traceback message is:

dert2@ma0phd201803:~$ python laplace_approx.py 
laplace_approx.py:12: RuntimeWarning: divide by zero encountered in log
    lpost = llikelihood + np.log(norm.pdf(mu, 0, 100)) + np.log(lognorm.pdf(sigma, loc= 0, s = 4))
laplace_approx.py:12: RuntimeWarning: divide by zero encountered in log
    lpost = llikelihood + np.log(norm.pdf(mu, 0, 100)) + np.log(lognorm.pdf(sigma, loc= 0, s = 4))
laplace_approx.py:12: RuntimeWarning: divide by zero encountered in log
    lpost = llikelihood + np.log(norm.pdf(mu, 0, 100)) + np.log(lognorm.pdf(sigma, loc= 0, s = 4))
/opt/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:83:     RuntimeWarning: invalid value encountered in reduce
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)

Answer 1

Some quick stepping in the debugger reveals that norm.pdf(y, mu, sigma) returns a nan array on one iteration. On that same iteration, lognorm.pdf(sigma, loc=0, s=4) returns zero. Even before that, sigma was assigned to a negative value. This is really where the problem starts. It would take some effort to track exactly why the function gets called with a negative params[1] value, but that does happen. I would use a debugger to step through the minimize function to see what is going on. I believe the negative value is passed on the 9th call to the function, if that helps. It is possible that the method that is being used to optimize the function does not work well for your particular function. You could try to use a different method, chosen by assigning to the method parameter (see the documentation for more details).