[英]What is the problem with my implementation of the cross-entropy function?
我正在学习神经网络,我想在 python 中编写一个函数cross_entropy
。 它被定义为
其中N
是样本数, k
是类别数, log
是自然对数,如果样本i
属于第j
类,则t_i,j
为 1,否则为0
,而p_i,j
是样本i
所在的预测概率j
级。 为了避免对数的数值问题,将预测剪裁到[10^{−12}, 1 − 10^{−12}]
范围。
根据上面的描述,我通过将预测裁剪到[epsilon, 1 − epsilon]
范围来写下代码,然后根据上面的公式计算 cross_entropy。
def cross_entropy(predictions, targets, epsilon=1e-12):
"""
Computes cross entropy between targets (encoded as one-hot vectors)
and predictions.
Input: predictions (N, k) ndarray
targets (N, k) ndarray
Returns: scalar
"""
predictions = np.clip(predictions, epsilon, 1. - epsilon)
ce = - np.mean(np.log(predictions) * targets)
return ce
下面的代码将用于检查函数cross_entropy
是否正确。
predictions = np.array([[0.25,0.25,0.25,0.25],
[0.01,0.01,0.01,0.96]])
targets = np.array([[0,0,0,1],
[0,0,0,1]])
ans = 0.71355817782 #Correct answer
x = cross_entropy(predictions, targets)
print(np.isclose(x,ans))
上面代码的输出是 False,也就是说我定义函数cross_entropy
代码不正确。 然后我打印cross_entropy(predictions, targets)
。 它给出了0.178389544455
,正确的结果应该是ans = 0.71355817782
。 有人可以帮我检查一下我的代码有什么问题吗?
您根本没有那么远,但请记住,您正在取 N 个总和的平均值,其中 N = 2(在本例中)。 所以你的代码可以是:
def cross_entropy(predictions, targets, epsilon=1e-12):
"""
Computes cross entropy between targets (encoded as one-hot vectors)
and predictions.
Input: predictions (N, k) ndarray
targets (N, k) ndarray
Returns: scalar
"""
predictions = np.clip(predictions, epsilon, 1. - epsilon)
N = predictions.shape[0]
ce = -np.sum(targets*np.log(predictions+1e-9))/N
return ce
predictions = np.array([[0.25,0.25,0.25,0.25],
[0.01,0.01,0.01,0.96]])
targets = np.array([[0,0,0,1],
[0,0,0,1]])
ans = 0.71355817782 #Correct answer
x = cross_entropy(predictions, targets)
print(np.isclose(x,ans))
在这里,我认为如果你坚持使用np.sum()
更清楚一些。 另外,我在np.log()
添加了 1e-9 以避免在计算中出现 log(0) 的可能性。 希望这可以帮助!
注意:根据@Peter 的评论,如果您的 epsilon 值大于0
,则1e-9
的偏移量确实是多余的。
def cross_entropy(x, y):
""" Computes cross entropy between two distributions.
Input: x: iterabale of N non-negative values
y: iterabale of N non-negative values
Returns: scalar
"""
if np.any(x < 0) or np.any(y < 0):
raise ValueError('Negative values exist.')
# Force to proper probability mass function.
x = np.array(x, dtype=np.float)
y = np.array(y, dtype=np.float)
x /= np.sum(x)
y /= np.sum(y)
# Ignore zero 'y' elements.
mask = y > 0
x = x[mask]
y = y[mask]
ce = -np.sum(x * np.log(y))
return ce
def cross_entropy_via_scipy(x, y):
''' SEE: https://en.wikipedia.org/wiki/Cross_entropy'''
return entropy(x) + entropy(x, y)
from scipy.stats import entropy, truncnorm
x = truncnorm.rvs(0.1, 2, size=100)
y = truncnorm.rvs(0.1, 2, size=100)
print np.isclose(cross_entropy(x, y), cross_entropy_via_scipy(x, y))
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.