在Lex中使用Regex中的Unicode Range作为规则

Question

import re
import ply.lex as lex

#rest of the code

def t_WORD(t): #WORD is a token defined in the tokens tuple
    r'[\u0C80-\u0CFF]+'
    #rest of the actions

This snippet provides an error stating illegal character. All characters are in the unicode range specified in the regex rule.

What can be the problem? Thanks in advance.

Answer 1

The lexer should work properly with both Unicode given as token and pattern matching rules. If you need to supply optional flags to the re.compile() function, use the reflags option to lex.

lex.lex(reflags=re.UNICODE)

As alternative, see How to validate kannada words and Python Lex-Yacc