Keras model.predict給model.evalute提供了不同的結果

Question

我正在訓練Keras LSTM進行命名實體識別。 單詞和字符級別都有一個雙向LSTM。

在訓練過程中，訓練和測試裝置的准確性都很高。 訓練后，我可以在測試集上運行model.evaluate()並獲得高達99％的准確率。

但是，如果我在X_test上使用model.predict() ，則該模型僅預測每個序列是零數組（我用於填充），並且精度為60％，f1得分為0。這是編碼：

class BiLSTM:

    def __init__(self):
        self.annotations = load_annotations('data/annotations.p')
        self.cache = format_data(self.annotations)
        self.cv_sets = cross_val_sets(self.cache['padded_sents'],     self.cache['padded_labels'])
        self.embedding_matrix = get_embedding_matrix(self.cache['word_to_integer'])
        self.char_cache = format_char_data(self.annotations,
                                           self.cache['word_to_integer'].keys(),
                                           self.cache['max_sequence_length'])
        self.model = None

    def fit_model(self, X_train, y_train, X_char_train):

        #Extract parameters from the cache
        word_to_integer = self.cache['word_to_integer']
        n_words = self.cache['n_words']
        n_tags = self.cache['n_tags']
        max_sequence_length = self.cache['max_sequence_length']
        X_char = self.char_cache['X_char']
        max_len_char = self.char_cache['max_len_char']
        n_chars = self.char_cache['n_chars']

        # (among top max_features most common words)
        batch_size = 32


        #Word input
        word_in = Input(shape=(max_sequence_length,))

        # Word embedding matrix
        embedding_matrix = get_embedding_matrix(word_to_integer)

        # Word Embedding layer
        embedding_layer = Embedding(input_dim=n_words + 1,
                                output_dim=200,
                                weights=[embedding_matrix],
                                input_length=max_sequence_length,
                                trainable=False)(word_in)

        # input and embeddings for characters
        char_in = Input(shape=(max_sequence_length, max_len_char,))
        emb_char = TimeDistributed(Embedding(input_dim=n_chars + 2, output_dim=10,
                                         input_length=max_len_char, mask_zero=True))(char_in)

        # character LSTM to get word encodings by characters
        char_enc = TimeDistributed(LSTM(units=20, return_sequences=False,
                                    recurrent_dropout=0.5))(emb_char)

        # main LSTM
        x = concatenate([embedding_layer, char_enc])
        x = SpatialDropout1D(0.3)(x)
        main_lstm = Bidirectional(LSTM(units=50, return_sequences=True,
                                   recurrent_dropout=0.6))(x)
        out = TimeDistributed(Dense(n_tags + 1, activation="softmax"))(main_lstm)

        model = Model([word_in, char_in], out)
        optimizer = Adam(lr=0.01, beta_1=0.9, beta_2=0.999)
        model.compile(optimizer=optimizer, loss="categorical_crossentropy", metrics=["acc"])


        model.fit([X_train, X_char_train], y_train,
                        batch_size=32, epochs=5, validation_split=0.2, verbose=1)

        self.model = model


    def run(self, cutoff = 0.8):
        sents = self.cache['padded_sents']
        labels = self.cache['padded_labels']
        # Train a model
        cutoff = int(sents.shape[0]*0.8)
        X_train = sents[:cutoff]
        X_test = sents[cutoff:]
        y_train = labels[:cutoff]
        y_test = labels[cutoff:]
        X_char_train = np.array(self.char_cache['X_char'])[:cutoff]
        X_char_test = np.array(self.char_cache['X_char'])[cutoff:]

        self.fit_model(X_train, y_train, X_char_train)

        # Accuracy metrics
        loss, accuracy = self.model.evaluate([X_test, X_char_test], y_test)

        print(accuracy)

        probs = self.model.predict([X_test, X_char_test])

        predicted = probs.argmax(axis=-1)
        actual = y_test.argmax(axis=-1)
        accuracy, f1 = get_metrics(actual, predicted, self.cache['integer_to_label'])
        print('acc: {}, f1: {}'.format(accuracy, f1))


if __name__ == "__main__":
    lstm = BiLSTM()
    lstm.run()

我已經進行了廣泛的搜索，找不到解決方案。 任何幫助表示贊賞，謝謝！

Answer 1

您是否嘗試過為單個序列打印出單個預測，例如

predicted = probs[0].argmax(axis = -1)

並與

actual = y_test[0].argmax(axis = -1)

比較此結果將有助於您進行調試。 如果此預測相同，則get_metrics必須存在一些問題。 試試這個並發布反饋。