您需要将 EOS 和 BOS 代币放入自动编码器转换器中吗？

Question

I'm starting to wrap my head around the transformer architecture, but there are some things that I am not yet able to grasp.

In decoder-free transformers, such as BERT, the tokenizer includes always the tokens CLS and SEP before and after a sentence. I understand that CLS acts both as BOS and as a single hidden output that gives the classification information, but I am a bit lost about why does it need SEP for the masked language modeling part.

I'll explain a bit more about the utility I expect to get. In my case, I want to train a transformer to act as an autoencoder, so target = input. There would be no decoder, since my idea is to reduce the dimensionality of the original vocabulary into less embedding dimensions, and then study (not sure how yet, but will get there) the reduced space in order to extract useful information.

Therefore, an example would be:

string_input = "The cat is black" 
tokens_input =  [1,2,3,4]

string_target = "The cat is black"
tokens_output = [1,2,3,4]

Now when tokenizing, assuming that we tokenize in the basis of word by word, what would be the advantage of adding BOS and EOS?

I think these are only useful when you are using the self-attention decoder, right? so, since in that case, for the decoder the outputs would have to enter right-shifted, the vectors would be:

input_string = "The cat is black EOS"
input_tokens = [1,2,3,4,5]

shifted_output_string = "BOS The cat is black"
shifted_output_tokens = [6,1,2,3,4]

output_string = "The cat is black EOS"
output_token = [1,2,3,4,5]

However, BERT does not have a self-attention decoder, but a simple feedforward layer. That is why I'm not sure of understanding the purpose of these special tokens.

In summary, the questions would be:

• Do you always need BOS and EOS tokens, even if you don't have a transformer decoder?
• Why does BERT, that does not have a transformer decoder, require the SEP token for the masked language model part?

Answer 1

First, a little about BERT - BERT word embeddings allow for multiple vector representations for the same word, based on the context in which the word was used. In this sense, BERT embeddings are context-dependent . BERT explicitly takes the index position of each word in the sentence while calculating its embedding. The input to BERT is a sentence rather than a single word. This is because BERT needs the context of the whole sentence to determine the vectors of the words in the sentence. If you only input a single word vector to BERT it would completely defeat the purpose of BERT's bidirectional, contextual nature. The output is then a fixed-length vector representation of the whole input sentence. BERT provides support for out-of-vocabulary words because the model learns words at a “subword” level (also called “word-pieces” ).

The SEP token is used to help BERT differentiate between two different word sequences. This is necessary in next-sequence-prediction (NSP). CLS is also necessary in NSP to let BERT know when the first sequence begins. Ideally you would use a format like this:

CLS [sequence 1] SEP [sequence 2] SEP

Note that we are not using any BOS or EOS tokens. The standard BERT tokenizer does not include these. We can see this if we run the following code:

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
print(tokenizer.eos_token)
print(tokenizer.bos_token)
print(tokenizer.sep_token)
print(tokenizer.cls_token)

Output: None None [SEP] [CLS]

For masked-language-modeling (MLM), we are only concerned with the MASK token, since the model's objective is merely to guess the masked token.

BERT was trained on both NSP and MLM and it is the combination of those two training methods that make BERT so effective.

So to answer your questions - you do not "always need" EOS and/or BOS. In fact, you don't "need" them at all. However, if you are fine-tuning BERT for a specific downstream task, where you intent to use BOS and EOS tokens (the manner of which, is up to you), then yes I suppose you would include them as special tokens. But understand that BERT was not trained with those in mind and you may see unpredictable/unstable results.