[英]logistic / sigmoid function implementation numerical precision
in scipy.special.expit
, logistic function is implemented like the following: 在
scipy.special.expit
,logistic函数实现如下:
if x < 0
a = exp(x)
a / (1 + a)
else
1 / (1 + exp(-x))
However, I have seen implementations in other languages/frameworks that simply do 但是,我已经看到了其他语言/框架中的实现
1 / (1 + exp(-x))
I am wondering how much benefit the scipy version actually brings. 我想知道scipy版本实际带来了多少好处。
For very small x
, the result approaches to 0. It works even if exp(-x)
overflows to Inf
. 对于非常小的
x
,结果接近0.即使exp(-x)
溢出到Inf
它也能工作。
It's really just for stability - putting in values that are very large in magnitude might return unexpected results otherwise. 它实际上只是为了稳定性 - 放入幅度非常大的值可能会返回意外的结果。
If expit
was implemented just as 1 / (1 + exp(-x))
then putting a value of -710
into the function would return nan
, whereas -709
would give a value close to zero as expected. 如果
expit
开始实施,就像1 / (1 + exp(-x))
然后将值-710
到函数将返回nan
,而-709
会如预期接近于零给出一个值。 This is because exp(710)
is too big to be a double. 这是因为
exp(710)
太大而不能成为双倍。
The branching in the code just means that this scenario is avoided. 代码中的分支只意味着避免了这种情况。
See also this question and answer on Stack Overflow. 另请参阅Stack Overflow上的这个问题和答案 。
Seems it would be more efficient to use: 似乎使用效率更高:
if x < -709
sigmoid = 0.0
else
sigmoid = 1.0 / (1.0 + exp(-x))
unless you need number with 10^-309 precision (see below) which seems overkill! 除非你需要10 ^ -309精度的数字(见下文),这似乎有点过分!
>>> 1 / (1 + math.exp(709.78))
5.5777796105262746e-309
Another way to do it would be 另一种方法是
python np.where(x > 0, 1. / (1. + np.exp(-x)), np.exp(x) / (np.exp(x) + np.exp(0)))
Since np.exp(x) / (np.exp(x) + np.exp(0))
is equivalent to 1. / (1. + np.exp(-x))
but more stable for negative values 由于
np.exp(x) / (np.exp(x) + np.exp(0))
相当于1. / (1. + np.exp(-x))
但对负值更稳定
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.