为什么我不能实现解码函数到字符串？

Question

I'm researching a dataset and rerun my coworker's code. When tokenizing text data, the code shown below doesn't work on my macbook, however, worked well in my coworker's computer.Here is the code.

I don't know which version does he have but mine is python3.6. Is it the problem of different versions?

s=title+' '+author+' '+text
 tokens=word_tokenize(s.decode('ascii','ignore').lower())

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-e50403f82604> in <module>
     10         flushPrint(m/100)#208
     11     s=title+' '+author+' '+text
---> 12     tokens=word_tokenize(s.decode('ascii','ignore').lower())
     13     tokens = [z for z in tokens if not z in stopset and len(z)>1]
     14     k=[]

AttributeError: 'str' object has no attribute 'decode'

Answer 1

The issue is most probably due to the changes between python2 and python3

In python2

• '' is of type str and thus supports ''.decode()
• u'' is of type unicode and thus supports u''.encode()

In python3 this is reversed

• '' is of type unicode and thus supports ''.encode()
• u'' is of type byte and thus supports u''.decode()

So in your case, depending on the type of your variables you might have to do something like

s = title + b' ' + author + b' ' + text

of just resort to python 2 :)