如何在 Python 上使用正则表达式在不忽略括号的情况下标记文本

Question

How to tokenize the text without ignoring their parenthesis and the () using regex?

For example:

I want to tokenize this sentence:

I don't like to eat Cici's food (it is true).

I used this regex:

pattern = r'''(?x)([A-Z]\.)+|\w+(-\w+)*|\$?\d+(\.\d+)?%?|\.\.\.|[][.,;"'?():-_`]'''
tokenize_list = nltk.regexp_tokenize(sentence, pattern)

But the output is not like what I want:

I
don
'
t
like
to
eat
Cici
'
s
food
(
it
is
true
)
.

The output that I want should be like this , which is consider the parenthesis and not tokenize ( and a word after it and also not tokenize ) and a word before it:

I
don't
like
to
eat
Cici's
food
(it
is
true)
.

Anyone can helps me? Thank you.

Answer 1

You can use a regex like this:

(['()\w]+|\.)

Working demo

Match information

MATCH 1
1.  [0-1]   `I`
MATCH 2
1.  [2-7]   `don't`
MATCH 3
1.  [8-12]  `like`
MATCH 4
1.  [13-15] `to`
MATCH 5
1.  [16-19] `eat`
MATCH 6
1.  [20-26] `Cici's`
MATCH 7
1.  [27-31] `food`
MATCH 8
1.  [32-35] `(it`
MATCH 9
1.  [36-38] `is`
MATCH 10
1.  [39-44] `true)`
MATCH 11
1.  [44-45] `.`