Using keras for time series classification with varied sized sequences

Question

I want to make a neural network, which can take as an input a number of coordinates of body parts (extracted using posenet) which can then classify which exercise the user is doing. To do this i wanted to give my network a series of coordinates and classify 7 different exercises, with one of them being idle. I have loaded the training data into a list of 6 elements with each element being a numpy array of variable length (one being bout 12.000 and down to about 2.000) with each time step consisting of 51 values (17 bodyparts with an x and y coordinate and a confidence value from posenet). My model looks like this:

    model.add(LSTM(32, input_shape=(None, 51), return_sequences=True))
    model.add(LSTM(16, return_sequences=True))
    model.add(LSTM(8, return_sequences=True))
    model.add(Dense(7, activation='softmax'))

    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

I try to fit the model using this:

model.fit(train_x, train_y,
          validation_data=(validation_x, validation_y), epochs=100,
          callbacks=[EarlyStopping(monitor='val_loss')])

And i get the following error:

ValueError: Data cardinality is ambiguous:
  x sizes: 12578, 3102, 12684, 7021, 2111, 2635
  y sizes: 12578, 3102, 12684, 7021, 2111, 2635
Make sure all arrays contain the same number of samples.

The shapes of my data are as follows:

train_x: (12578, 51), (3102, 51), (12684, 51), (7021, 51), (2111, 51), (2635, 51)
train_y: (12578,), (3102,), (12684,), (7021,), (2111,), (2635,)
validation_x: (2989, 51), (4425, 51), (2919, 51)
validation_y: (2989,), (4425,), (2919,)

How do i format the input to be able to train my model?

Answer 1

Many solutions can solve your problem but I will focus on two different approaches that I use the most:

Generate a fixed window as input

You can build a fixed window of size (ig n_steps= 50 ) and the input size will be n_steps . This is a function that I use to build this kind of fixed window size approach from the series:

def split_sequence(sequence, n_steps,pred_steps):
    X, y = list(), list()
    for i in range(len(sequence)):
        # find the end of this pattern
        end_ix = i + n_steps
        # check if we are beyond the sequence
        if end_ix > len(sequence)-pred_steps:
            break
        # gather input and output parts of the pattern
        seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:end_ix+pred_steps]
        X.append(seq_x)
        y.append(seq_y)
    return np.array(X), np.array(y)

with pred_steps being the future window prediction to be used as the target for the model. This function will give you, from a sequence the X and y for your model.

Give the whole series as input but fill the non-seen events with zeros

You can generate an input with the series size filled with zeros and append every new value at each time-step:

series = [1,2,3,4,5]
#at time t=0
input = [0,0,0,0,0]
#at time t=1
input = [0,0,0,0,1]
#at time t=2
input = [0,0,0,1,2]
# and this continue until the end of the series

the output will have a similar shape as in the fixed-size solution. To implement this solution the Numpy function np.delete and np.append or np.concatenate can be used.

Check Numpy documentation for that: https://numpy.org/doc/stable/reference/routines.array-manipulation.html

Usin only the last value of the series

Since you are using an LSTM architecture, with the memory gates, the previous outputs and inputs are remembered by the network. I am not sure about this approach, but it could also work.