Bert DL 模型错误：重塑的输入是具有 3200 个值的张量，但请求的形状具有 3328

Question

I am repeating the codes from

https://towardsdatascience.com/text-classification-with-nlp-tf-idf-vs-word2vec-vs-bert-41ff868d1794

Here is the code for BERT classifier. The error code is at the end of this question:

## distil-bert tokenizer
tokenizer = transformers.AutoTokenizer.from_pretrained('distilbert-base-uncased', do_lower_case=True)
dtf_train, dtf_test = train_test_split(all_ct_df_twoyear_v4, test_size=0.2,random_state=101)

y_train = dtf_train['isChange'].values
y_test = dtf_test['isChange'].values


corpus = dtf_train["comment"]
maxlen = 50

## add special tokens
maxqnans = np.int((maxlen-20)/2)
corpus_tokenized = ["[CLS] "+
             " ".join(tokenizer.tokenize(re.sub(r'[^\w\s]+|\n', '', 
             str(txt).lower().strip()))[:maxqnans])+
             " [SEP] " for txt in corpus]

## generate masks
masks = [[1]*len(txt.split(" ")) + [0]*(maxlen - len(
           txt.split(" "))) for txt in corpus_tokenized]
    
## padding
txt2seq = [txt + " [PAD]"*(maxlen-len(txt.split(" "))) if len(txt.split(" ")) != maxlen else txt for txt in corpus_tokenized]
    
## generate idx
idx = [tokenizer.encode(seq.split(" ")) for seq in txt2seq]
    
X_train = [np.asarray(idx, dtype='int32'), 
           np.asarray(masks, dtype='int32')] 
           #np.asarray(segments, dtype='int32')]
corpus = dtf_test["comment"]
maxlen = 50

## add special tokens
maxqnans = np.int((maxlen-20)/2)
corpus_tokenized = ["[CLS] "+
             " ".join(tokenizer.tokenize(re.sub(r'[^\w\s]+|\n', '', 
             str(txt).lower().strip()))[:maxqnans])+
             " [SEP] " for txt in corpus]

## generate masks
masks = [[1]*len(txt.split(" ")) + [0]*(maxlen - len(
           txt.split(" "))) for txt in corpus_tokenized]
    
## padding
txt2seq = [txt + " [PAD]"*(maxlen-len(txt.split(" "))) if len(txt.split(" ")) != maxlen else txt for txt in corpus_tokenized]
    
## generate idx
idx = [tokenizer.encode(seq.split(" ")) for seq in txt2seq]
    

## feature matrix
X_test = [np.asarray(idx, dtype='int32'), 
           np.asarray(masks, dtype='int32')]
           #np.asarray(segments, dtype='int32')]
## inputs
idx = layers.Input((50), dtype="int32", name="input_idx")
masks = layers.Input((50), dtype="int32", name="input_masks")
## pre-trained bert with config
config = transformers.DistilBertConfig(dropout=0.2, 
           attention_dropout=0.2)
config.output_hidden_states = False
nlp = transformers.TFDistilBertModel.from_pretrained('distilbert-base-uncased', config=config)
bert_out = nlp(idx, attention_mask=masks)[0]
## fine-tuning
x = layers.GlobalAveragePooling1D()(bert_out)
x = layers.Dense(64, activation="relu")(x)
y_out = layers.Dense(len(np.unique(y_train)), 
                     activation='softmax')(x)
## compile
model = models.Model([idx, masks], y_out)
for layer in model.layers[:3]:
    layer.trainable = False
model.compile(loss='sparse_categorical_crossentropy', 
              optimizer='adam', metrics=['accuracy'])
model.summary()
## encode y
dic_y_mapping = {n:label for n,label in 
                 enumerate(np.unique(y_train))}
inverse_dic = {v:k for k,v in dic_y_mapping.items()}
y_train = np.array([inverse_dic[y] for y in y_train])
## train
training = model.fit(x=X_train, y=y_train, batch_size=64, 
                     epochs=1, shuffle=True, verbose=1, 
                     validation_split=0.3)
## test
predicted_prob = model.predict(X_test)
predicted = [dic_y_mapping[np.argmax(pred)] for pred in 
             predicted_prob]

error:

    InvalidArgumentError:  Input to reshape is a tensor with 3200 values, but the requested shape has 3328
    [[node functional_45/tf_distil_bert_model_22/distilbert/transformer/layer_._0/attention/Reshape_3 (defined at X:\Users\xuanyu\Anaconda3\lib\site-packages\transformers\modeling_tf_distilbert.py:237) ]] [Op:__inference_train_function_287881]

    Errors may have originated from an input operation.
    Input Source operations connected to node 
    functional_45/tf_distil_bert_model_22/distilbert/transformer/layer_._0/attention/Reshape_3:
    functional_45/tf_distil_bert_model_22/distilbert/Cast (defined at 
    X:\Users\xuanyu\Anaconda3\lib\site- 
    packages\transformers\modeling_tf_distilbert.py:466)

    Function call stack:
    train_function

I have been searching and searching and tuning the parameters but still got me this error. I did not find anywhere to change how the reshaping size can be modified.

Answer 1

I hope my answer is not too late but for me it worked with the 2.1 transformers version. Execute

pip install transformers==2.1