[英]Optimize sparse softmax cross entropy with L2 regularization
I was training my network using tf.losses.sparse_softmax_cross_entropy
as the classification function in the last layer and everything was working fine. 我正在使用
tf.losses.sparse_softmax_cross_entropy
作为最后一层的分类函数来训练我的网络,并且一切工作正常。
I simply added a L2 regularization over my weights now and my loss is not getting optimized anymore. 我现在只是在权重上添加了L2正则化,并且损失不再得到优化。 What can be happening?
会发生什么事?
reg = tf.nn.l2_loss(w1) + tf.nn.l2_loss(w2)
loss = tf.reduce_mean(tf.losses.sparse_softmax_cross_entropy(y, logits)) + reg*beta
train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)
It is hard to answer with certainty given the provided information, but here is a possible cause: 给定提供的信息,很难确定答案,但这是可能的原因:
tf.nn.l2_loss
is computed as a sum over the elements, while your cross-entropy loss is reduced to its mean (cf tf.reduce_mean
), hence a numerical unbalance between the 2 terms. tf.nn.l2_loss
计算为元素上的总和,而交叉熵损失减少到其均值(cf tf.reduce_mean
),因此这两个项之间存在数值不平衡。
Try for instance to divide each L2 loss by the number of elements it is computed over (eg tf.size(w1)
). 例如,尝试将每个L2损耗除以计算
tf.size(w1)
的元素数量(例如tf.size(w1)
)。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.