顺序学习以进行语言翻译，看不见的单词怎么办

Question

sequence to sequence learning is a powerful mechanism for language translation, especially using it locally in a context specific case.

I am following this pytorch tutorial for the task.

However, the tutorial did not split the data into training and testing. You might think its not a big deal, just split it up, use one chunk for training and the other for testing. But it is not that simple.

Essentially, the tutorial creates the indices of the seen words while leading the dataset. The indices are simply stored in the dictionary. This is before going to the encoder RNN, just a simple conversion kind of task from words to the numbers.

If data is split up at random, what happens is, one of the keyword may not appear in the sentences from the training set, and so may not have an index at all. If it shows up at the time of testing, what should be done?

Extend the dictionary?

Answer 1

Sequence to sequence models performance strongly depend on count of unique words in vocabulary. Each unique word has to be encountered a number of times in training set, such that model can learn it correct usage. Words that appears few times cannot be used by the model, as model can't learn enough information about such words. In practice, the size of the dictionary is usually reduced, replacing the rare words with a special "UNK" token. Therefore, if a new word occurs during testing, it can be assumed that it is rare (since it never appears in the training set) and replace it with "UNK".