[英]tokenize according space and punctuation, punctuation kept
我正在寻找一种解决方案,以根据空格或标点符号化或拆分。 仅标点必须保留在结果中。 它将用于识别语言(python,java,html,c ...)
输入string
可以是:
class Foldermanagement():
def __init__(self):
self.today = invoicemng.gettoday()
...
我期望的输出是列表/标记,如下所述:
['class', 'Foldermanagement', '(', ')', ':', 'def', '_', '_', 'init', ... ,'self', '.', 'today', '=', ...]
欢迎任何解决方案,谢谢。
我认为这是您要寻找的东西:
import string, re, itertools
text = """
class Foldermanagement():
def __init__(self):
self.today = invoicemng.gettoday()
"""
separators = string.punctuation + string.whitespace
separators_re = "|".join(re.escape(x) for x in separators)
tokens = zip(re.split(separators_re, text), re.findall(separators_re, text))
flattened = itertools.chain.from_iterable(tokens)
cleaned = [x for x in flattened if x and not x.isspace()]
# ['class', 'Foldermanagement', '(', ')', ':', 'def', '_', '_',
# 'init', '_', '_', '(', 'self', ')', ':', 'self', '.', 'today', '=',
# 'invoicemng', '.', 'gettoday', '(', ')']
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.