NLP 使用替换令牌

Question

I read a lot of articles that deal with different NLP classification tasks and I saw that most of them specify in the pre-processing section that they use replacement tokens:

eg We removed and replaced the URLs, emojis and punctuation with replacement tokens: <URL>, <EMOJI>, <PUNCT> .

I am quite new to this domain and I was wondering if there is some special way to deal with this kind of tokens/tags? Is it necessary to use < > or is this just a way to signal this replacement and for helping the classifier in finding a pattern?

Any help would be greatly appreciated.

Answer 1

From what I did, in the pre-processing section, people replace all tokens (chars, morphemes, words) with numbers. These replacement tokens are nothing but numbers as well, <URL> is just a way to present it to humans.