如何使用Keras实施注意力

Question

I have a keras model that has sequence of inputs and sequence of outputs where each input has an associated output(Label).

model = Sequential()
model.add(Masking(mask_value=5, input_shape= (Seq_in.shape[1],1)))
model.add(Bidirectional(LSTM(256,  return_sequences=True)))
model.add(Dropout(0.2))
model.add(Bidirectional(LSTM(256, return_sequences=True))) 
model.add(Dropout(0.2))
model.add(Dense(n_Labels, activation='softmax'))  # n_Labels is the number of labels which is 15
sgd = optimizers.SGD(lr=.1,momentum=0.9,decay=1e-3,nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
model.fit(X_train,Y_train,epochs=2, validation_data=(X_val, Y_val),verbose=1)

Now I want to implement attention mechanism following this work Zhou et al: "Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification".

For each output, we compute the tanh for each output states in the sequence(Eq# 9 in the paper), then we compute the softmax for each output states in regard to the current output (Eq#10), then we multiply each output states with the corrsponding softmax (attention) (Eq# 11), then we take the sum of the weighted states, then we take the tanh of the final output which represent. Finally we concatenate the attention vector with the output states.

How can I do that? is it possiple to that with the keras API or I have to come up with my own custom layer? any help ?

Thank you in advance....

Answer 1

There is no Keras API yet. But a lot of hard working programmers made some good implementations using Keras. You can try looking at the code in keras-monotonic-attention .