弄乱我的unicode输出-但是在哪里以及如何进行？

Question

I am doing a word count on some text files, storing the results in a dictionary. My problem is that after outputting to file, the words are not displayed right even if they were in the original text. (I use TextWrangler to look at them). For instance, dashes show up as dashes in the original but as \— in the output; in the output, very word is prefixed by a u as well.

Problem

I do not know where, when and how in my script this happens.

I am reading the file with codecs.open() and outputting them with codecs.open() and as json.dump() . They both go wrong in the same way. In between, all is do is

1. tokenizing

2. regular expressions

3. collect in dictionary

And I don't know where I mess things up; I have de-activated tokenizing and most other functions to no effect. All this is happening in Python 2. Following previous advice, I tried to keep everything within the script in Unicode.

Here is what I do (non-relevant code omitted):

#read in file, iterating over a list of "fileno"s
with codecs.open(os.path.join(dir,unicode(fileno)+".txt"), "r", "utf-8") as inputfili:
            inputtext=inputfili.read()

#process the text: tokenize, lowercase, remove punctuation and conjugation
content=regular expression to extract text w/out metadata
contentsplit=nltk.tokenize.word_tokenize(content)
text=[i.lower() for i in contentsplit if not re.match(r"\d+", i)]
text= [re.sub(r"('s|s|s's|ed)\b", "", i) for i in text if i not in string.punctuation]

#build the dictionary of word counts
for word in text:
    dicti[word].append(word)

#collect counts for each word, make dictionary of unique words
dicti_nos={unicode(k):len(v) for k,v in dicti.items()}
hapaxdicti= {k:v for k,v in perioddicti_nos.items() if v == 1}

#sort the dictionary
sorteddict=sorted(dictionary.items(), key=lambda x: x[1], reverse=True)

#output the results as .txt and json-file
with codecs.open(file_name, "w", "utf-8") as outputi:
    outputi.write("\n".join([unicode(i) for i in sorteddict]))
with open(file_name+".json", "w") as jsonoutputi:
    json.dump(dictionary, jsonoutputi,  encoding="utf-8")

EDIT: Solution

Looks like my main issue was writing the file in the wrong way. If I change my code to what's reproduced below, things work out. Looks like joining a list of (string, number) tuples messed the string part up; if I join the tuples first, things work.

For the json output, I had to change to codecs.open() and set ensure_ascii to False . Apparently just setting the encoding to utf-8 does not do the trick like I thought.

with codecs.open(file_name, "w", "utf-8") as outputi:
    outputi.write("\n".join([":".join([i[0],unicode(i[1])]) for i in sorteddict]))

with codecs.open(file_name+".json", "w", "utf-8") as jsonoutputi:
    json.dump(dictionary, jsonoutputi,  ensure_ascii=False)

Thanks for your help!

Answer 1

As your example is partially pseudocode there's no way to run a real test and give you something that runs and has been tested, but from reading what you have provided I think you may misunderstand the way Unicode works in Python 2.

The unicode type (such as is produced via the unicode() or unichr() functions) is meant to be an internal representation of a Unicode string that can be used for string manipulation and comparison purposes. It has no associated encoding. The unicode() function will take a buffer as its first argument and an encoding as its second argument and interpret that buffer using that encoding to produce an internally usable Unicode string that is from that point forward unencumbered by encodings.

That Unicode string isn't meant to be written out to a file; all file formats assume some encoding, and you're supposed to provide one again before writing that Unicode string out to a file. Everyplace you have a construct like unicode(fileno) or unicode(k) or unicode(i) is suspect both because you're relying on a default encoding (which probably isn't what you want) and because you're going on to expose most of these values directly to the file system.

After you're done working with these Unicode strings you can use the built-in method encode() on them with your desired encoding as an argument to pack them into strings of ordinary bytes set as required by your encoding.

So looking back at your example above, your inputtext variable is an ordinary string containing data encoded per the UTF-8 encoding. This isn't Unicode. You could convert it to a Unicode string with an operation like inputuni = unicode(inputtext, 'utf-8') and operate on it like that if you chose, but for what you're doing you may not even find it necessary. If you did convert it to Unicode though you'd have to perform the equivalent of a inputuni.encode('UTF-8') on any Unicode string that you were planning on writing out to your file.