您可以使用字典（文本）进行正则表达式标记化吗？

Question

I was wondering if we can use a text files as a means for tokenization. For example let's say there is a file(dictionary) and you want to tokenize you check first dictionary to tokenize.

Eg:

Dict_list = [environment test, apple cat, test rest]

Text : The environment test is the best apple in the world apple cat is the test rest.

Assume the text list is big and dict is also big, so if we want to tokenize it would tokenize by spaces however I need to tokenize whole text however I want to check dict_list to see if that should be one token.

so the token should be:

Token : "The", "environment test", "is", "the", "best apple", "in", "the", "world", "apple cat", "is", "the", "test rest".

I hope this makes sense.

Thank you in advance.

Answer 1

With nltk.tokenize package you can easily do this. For example:

>>> tokenizer.tokenize('Testing testing testing one two three'.split())
['Testing', 'testing', 'testing', 'one', 'two', 'three']

>>> tokenizer = MWETokenizer([('hors', "d'oeuvre")], separator='+')
>>> tokenizer.tokenize("An hors d'oeuvre tonight, sir?".split())
['An', "hors+d'oeuvre", 'tonight,', 'sir?']

Answer 2

This is one way but a workaround:

Python3 version:

from nltk.tokenize import regexp_tokenize

sent = "I like apple fruit but grape fruit more"
dict_list = ["apple fruit", "grape fruit"]
newdict = {}
for item in dict_list:
    dk = item.replace(" ", "_")
    newdict[item] = dk

for key, val in newdict.items():
    if key in sent:
        sent = sent.replace(key, val)

res = regexp_tokenize(sent, pattern='\S+')
print(res)

Output:

['I', 'like', 'apple_fruit', 'but', 'grape_fruit', 'more']

You can then replace all the underscores with space if you wish.