为什么 CNN model 的损失在整个 epoch 中变化不大？

Question

I am running a deep learning project about visual speech recognition tasks, and I found a strange phenomenon. In the first several epochs, the loss can decrease at a normal speed. I mean, in a epoch, the loss would decrease with the increase of the number of iterations. While in the later epochs, the loss almost unchanged in the whole epoch, but would decrease at the start of the next epoch.

Sometimes I interrupted the running code after an epoch finishing and restart from the trained weights. The loss would decrease at the start of the next epoch, too.

This is the training code:

    for epoch in range(283,args.epochs):

        model.train()
        running_loss, running_corrects, running_all, cer = 0., 0., 0., 0.

        for batch_idx, sample_batched in enumerate(dset_loaders['train']):
            optimizer.zero_grad()
            inputs,targets,lengths,y_lengths,idx = sample_batched
            inputs = inputs.float()     
            inputs, targets = inputs.to(device) , targets.to(device) 
            outputs = model(inputs)  
            loss = criterion(F.log_softmax(outputs,dim=-1),targets,lengths,y_lengths)
            loss.backward()
            optimizer.step()

            decoded = decoder.decode_greedy(outputs,lengths)
            cursor, gt = 0, []
            for b in range(inputs.size(0)):
                y_str = ''.join([vocabularies[ch] for ch in targets[cursor: cursor + y_lengths[b]]])
                gt.append(y_str)
                cursor += y_lengths[b]
            CER = decoder.cer_batch(decoded,gt)
            cer += CER
            cer_mean = cer/(batch_idx+1)

            running_loss += loss.data * inputs.size(0)
            running_all += len(inputs)
            if batch_idx == 0:
                since = time.time()      
            else (batch_idx+1) % args.interval == 0 or (batch_idx == len(dset_loaders['train'])-1):            
                print('Process: [{:5.0f}/{:5.0f} ({:.0f}%)]\tLoss: {:.4f}\tcer:{:.4f}\tCost time:{:5.0f}s\tEstimated time:{:5.0f}s\t'.format(
                    running_all,
                    len(dset_loaders['train'].dataset),
                    100. * batch_idx / (len(dset_loaders['train'])-1),
                    running_loss / running_all,
                    cer_mean,
                    time.time()-since,
                    (time.time()-since)*(len(dset_loaders['train'])-1) / batch_idx - (time.time()-since)))
        print('{} Epoch:\t{:2}\tLoss: {:.4f}\tcer:{:.4f}\t'.format(
            'pretrain',
            epoch,
            running_loss / len(dset_loaders['train'].dataset),
            cer_mean)+'\n')
        torch.save(model.state_dict(), save_path+'/'+args.mode+'_'+str(epoch+1)+'.pt')

I get very confused about this phenomenon. I think if the Loss hasn't changed in the whole epoch, the Loss in the next epoch shouldn't have changed either. Why the Loss still change at the beginning of the next epoch after unchanged in the whole epoch? Can someone help me solve this problem? Thanks!

Answer 1

I think it may be related to how you print the loss.

You have a running_loss that denotes the total loss for every data point calculated in this epoch and a running_all demote the total number of data points calculated in this epoch. You print the running_loss / running_all which is the average loss for every data point in this epoch.

As more data point is collected, even if the loss is steadily decreasing, the new loss is averaged with a larger number of the previously computed loss, which make the decreasing seems slower. Explained here: https://gist.github.com/valkjsaaa/b0b26075174a87b3fd302b4b52ab035a

I would suggest replace running_loss / running_all with loss.data / len(inputs) , which is the loss for this current batch and see if it would help.

The changed code should look like the following:

   for epoch in range(283,args.epochs):

        model.train()
        running_loss, running_corrects, running_all, cer = 0., 0., 0., 0.

        for batch_idx, sample_batched in enumerate(dset_loaders['train']):
            optimizer.zero_grad()
            inputs,targets,lengths,y_lengths,idx = sample_batched
            inputs = inputs.float()     
            inputs, targets = inputs.to(device) , targets.to(device) 
            outputs = model(inputs)  
            loss = criterion(F.log_softmax(outputs,dim=-1),targets,lengths,y_lengths)
            loss.backward()
            optimizer.step()

            decoded = decoder.decode_greedy(outputs,lengths)
            cursor, gt = 0, []
            for b in range(inputs.size(0)):
                y_str = ''.join([vocabularies[ch] for ch in targets[cursor: cursor + y_lengths[b]]])
                gt.append(y_str)
                cursor += y_lengths[b]
            CER = decoder.cer_batch(decoded,gt)
            cer += CER
            cer_mean = cer/(batch_idx+1)

            running_loss += loss.data * inputs.size(0)
            running_all += len(inputs)
            if batch_idx == 0:
                since = time.time()      
            else (batch_idx+1) % args.interval == 0 or (batch_idx == len(dset_loaders['train'])-1):            
                print('Process: [{:5.0f}/{:5.0f} ({:.0f}%)]\tLoss: {:.4f}\tcer:{:.4f}\tCost time:{:5.0f}s\tEstimated time:{:5.0f}s\t'.format(
                    running_all,
                    len(dset_loaders['train'].dataset),
                    100. * batch_idx / (len(dset_loaders['train'])-1),
                    loss.data,
                    cer_mean,
                    time.time()-since,
                    (time.time()-since)*(len(dset_loaders['train'])-1) / batch_idx - (time.time()-since)))
        print('{} Epoch:\t{:2}\tLoss: {:.4f}\tcer:{:.4f}\t'.format(
            'pretrain',
            epoch,
            running_loss / len(dset_loaders['train'].dataset),
            cer_mean)+'\n')
        torch.save(model.state_dict(), save_path+'/'+args.mode+'_'+str(epoch+1)+'.pt')