I need to remove punctuation from a unicode string. I've read a few posts and the most recommended one was this one .
I've implemented the following:
table = dict.fromkeys(i for i in range(sys.maxunicode) if unicodedata.category(chr(i)).startswith('P'))
def tokenize(message):
message = unicode(message,'utf-8').lower()
#print message
message = remove_punctuation_unicode(message)
return message
def remove_punctuation_unicode(string):
return string.translate(table)
But when I run the code, this error pops up:
table = dict.fromkeys(i for i in range(sys.maxunicode) if unicodedata.category(chr(i)).startswith('P'))
TypeError: must be unicode, not str
I can't quite figure it out what to do. Can someone tell me how to fix this?
Try unichr
instead of chr
:
Python 2.7.10 (default, Oct 14 2015, 16:09:02)
[GCC 5.2.1 20151010] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, unicodedata
>>> table = dict.fromkeys(i for i in range(sys.maxunicode) if unicodedata.category(unichr(i)).startswith('P'))
>>>
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.