utf-8在列表中搜索单词

Question

I have a lookup list generated from utf-8 file by

with open('stop_word_Tiba.txt') as f:
    newStopWords= list(itertools.chain( line.split() for line in f)) #save the file as list of lines
newStopWords1d=list(itertools.chain(*newStopWords)) # convert 2d list to 1d list

when I open the file I see that the word 'الو' is in there. so it is in the list, but the list now looks like ['\\xd8\\xa7\\xd9\\x84\\xd9\\x88', '\\xd8\\xa3\\xd9\\x84\\xd9\\x88', '\\xd8\\xa7\\xd9\\x88\\xd9\\x83\\xd9\\x8a', '\\xd8\\xa7\\xd9\\x84', '\\xd8\\xa7\\xd9\\x87', '\\xd8\\xa3\\xd9\\x87', '\\xd9\\x87\\xd9\\x84\\xd9\\x88', '\\xd8\\xa3\\xd9\\x88\\xd9\\x83\\xd9\\x8a', '\\xd9\\x88']

Then I would like to search if a specific word is in newStopWords1d the word 'الو' is '\\xd8\\xa7\\xd9\\x84\\xd9\\x88'

word='الو'
for w in newStopWords1d:
    if word == w.encode("utf-8"):
        print 'found'

The word is not found, I tried

    if word in newStopWords1d:
        print 'found'

but again the word is not seen. It seems like the problem with encoding but I couldn't solve it. can you please help me.

Answer 1

Worth mentioning was that you use Python 2.7.

word='الو'
for w in newStopWords1d:
    if word == w.decode("utf-8"):
        print 'found'

Even better solution is to use either the open function from io

import io

with io.open('stop_word_Tiba.txt', encoding="utf-8") as f:
    ...

or codecs module

import codecs

with codecs.open('stop_word_Tiba.txt', encoding="utf-8") as f:
    ...

as the built-in open function in Python 2.7 doesn't support specifying the encoding.

Answer 2

The problem solved by editing the open file statement as

with codecs.open("stop_word_Tiba.txt", "r", "utf-8") as f:
    newStopWords= list(itertools.chain( line.split() for line in f)) #save the file as list of lines
newStopWords1d=list(itertools.chain(*newStopWords))
    for w in newStopWords1d:
            if word.encode("utf-8") == w.encode("utf-8") :  
                      return 'found'

Thanks for you..