如何训练具有可变 output 大小的神经网络

Question

I have a working CNN-LSTM model trying to predict keypoints of human bodyparts on videos.

Currently, I have four keypoints as labels right hand, left hand, head and pelvis. The problem is that on some frames I can't see the four parts of the human that I want to label, so by default I set those values to (0,0) (which is a null coordinate).

The problem that I faced was the model taking in account those points and trying to regress on them while being in a sequence.

Thus, I removed the (0,0) points in the loss calculation and the gradient retropropagation and it works much better.

The problem is that the Four points are still predicted, so I am trying to know by any means how to make it predict a variable number of keypoints.

I thought of adding a third parameter (is it visible?), but it will probably add some complexity and loose the model.

Answer 1

I think that you'll have to write a custom loss function that computes the loss between points only when the target coordinates are not null.

See PyTorch custom loss function on writing custom losses.

Something like:

def loss(outputs, labels):
    err = 0
    n = 0
    for xo, xt in zip(outputs, labels):
        if xt.values == torch.zeros(2):  # null coord
            continue
        err += torch.nn.functional.mse_loss(xo, xt)
        n += 1
    return (err / n)

This is pseudo-code only. An alternative form which will avoid the loop is to have an explicit binary vector (as suggested by @leleogere) that you can then multiply by the loss on each coordinate before reducing.