[英]How can I test a word2vec over development data?
In a computer assignment, it's requested to implement word2vec algorithm to generate dense vectors for some words using a neural network. 在计算机任务中,要求实现word2vec算法,以使用神经网络为某些单词生成密集向量。 I implemented the neural network and trained it over training data.
我实现了神经网络,并通过训练数据对其进行了训练。 First, how can I test it over the test data?
首先,如何在测试数据上对其进行测试? The question asks to draw a plot showing the perplexity for the training and test data during training (epochs).
该问题要求绘制一个图表,显示训练期间(时期)训练和测试数据的困惑性。 I can do that for the loss, which is something like this:
我可以为此做些损失,就像这样:
EPOCH: 0 LOSS: 27030.09155006593
EPOCH: 0 P_LOSS: 24637.964948774144
EPOCH: 0 PP: inf
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:121: RuntimeWarning: overflow encountered in double_scalars
EPOCH: 1 LOSS: 25349.086587261085
EPOCH: 1 P_LOSS: 22956.95998596929
EPOCH: 1 PP: inf
EPOCH: 2 LOSS: 24245.455581381622
EPOCH: 2 P_LOSS: 21853.32898008983
EPOCH: 2 PP: inf
EPOCH: 3 LOSS: 23312.976009712416
EPOCH: 3 P_LOSS: 20920.849408420647
Which I gained by the following code: 我是通过以下代码获得的:
# CYCLE THROUGH EACH EPOCH
for i in range(0, self.epochs):
self.loss = 0
self.loss_prob = 0
# CYCLE THROUGH EACH TRAINING SAMPLE
for w_t, w_c in training_data:
# FORWARD PASS
y_pred, h, u = self.forward_pass(w_t)
# CALCULATE ERROR
EI = np.sum([np.subtract(y_pred, word) for word in w_c], axis=0)
# BACKPROPAGATION
self.backprop(EI, h, w_t)
# CALCULATE LOSS
self.loss += -np.sum([u[word.index(1)] for word in w_c]) + len(w_c) * np.log(np.sum(np.exp(u)))
self.loss_prob += -2*np.log(len(w_c)) -np.sum([u[word.index(1)] for word in w_c]) + (len(w_c) * np.log(np.sum(np.exp(u))))
print('EPOCH:',i, 'LOSS:', self.loss)
print('EPOCH:',i, 'P_LOSS:', self.loss_prob)
print('EPOCH:',i, 'PP:', 2**self.loss_prob)
However, I don't know how to find the perplexity for the training and development data in each epoch. 但是,我不知道如何找到每个时期的培训和发展数据的困惑。 Based on this question , it's said perplexity is
2**loss
. 基于这个问题 ,据说困惑是
2**loss
。 However, as I tried this formula, I gained INF
. 但是,当我尝试此公式时,获得了
INF
。 Could you guide me how can I calculate perplexity? 您能指导我如何计算困惑吗? Can I do it in the current code, or should I apply a function over whole development data?
我可以在当前代码中执行此操作,还是应该对整个开发数据应用功能?
Indeed, 2**20000
overflows the float. 实际上,
2**20000
溢出了浮点数。 You should normalize the loss per example, ie, divide by your training data size. 您应该按照示例对损失进行归一化,即除以训练数据大小。 Back-propagation works even if you sum them because dividing by a constant does not change the derivative, but in this way, the loss is not invariant to data size.
即使对它们进行求和,反向传播也可以工作,因为除以常数不会更改导数,但是通过这种方式,损耗不会随数据大小不变。
You can solve the INF problem using decimal. 您可以使用十进制来解决INF问题。
import decimal
decimal.Decimal(2)**decimal.Decimal(loss)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.