在pytorch中实现似然训练损失

Question

I am trying to implement the Unlikelihood Training loss that was proposed in this research paper: NEURAL TEXT DEGENERATION WITH UNLIKELIHOOD TRAINING . This loss is an updated version of the negative log-likelihood loss (NLLLOSS).

The main idea of this loss is that it avoids unwanted tokens during the training process.

This is my code:

def NLLLoss(logs, targets, c, alpha=0.1):
    out = torch.zeros_like(targets, dtype=torch.float)
    for i in range(len(targets)):
        # out[i] = logs[i][targets[i]] # The original implementation
        out[i] = alpha * (1 - logs[i][c[i]]) * logs[i][targets[i]]
    return -out.sum()/len(out)

The commented line is the original NLLLoss implementation. This code well, but I was wondering, is this implementation correct?

Answer 1

No, log(1-x) does not equal 1 - log(x). I think what need is here .