在Python中打印tsv文件的内容（使用UTF-8）

Question

The code I have below works fine in a file I've named tsv_test.py:

import csv

class ReadUTF8():

    def unicode_csv_reader(self, utf8_data, dialect=csv.excel_tab, **kwargs):
        csv_reader = csv.reader(utf8_data, dialect=dialect, **kwargs)
        for row in csv_reader:
            yield [unicode(cell, 'utf-8') for cell in row]


    def load_deck_data(self):
        filename = 'lexicon.tsv'
        reader = self.unicode_csv_reader(open(filename))
        for field1, field2, field3, field4 in reader:
            print field1, field2, field3, field4

ReadUTF8().load_deck_data()

But when I copy/paste it into my project (this is a kivy project), it breaks. Code and error below:

class StudyScreenManagement(ScreenManager):

    def unicode_csv_reader(self, utf8_data, dialect=csv.excel_tab, **kwargs):
        csv_reader = csv.reader(utf8_data, dialect=dialect, **kwargs)
        for row in csv_reader:
            yield [unicode(cell, 'utf-8') for cell in row]


    def load_deck_data(self):
        filename = 'lexicon.tsv'
        reader = self.unicode_csv_reader(open(filename))
        for field1, field2, field3, field4 in reader:
            print field1, field2, field3, field4

I doubt this is related, but just in case, the related .kv file:

Button:
    text: 'Lexicon'
    on_press: app.root.load_deck_data()

Output:

 File "/Users/bearnun/code/mingyu/mingyuKivy/mingyu_controllers.py", line 14, in load_deck_data
 for field1, field2, field3, field4 in reader:
 ValueError: need more than 1 value to unpack

::Side Note::

I tried just printing 'field1' in both cases. With that change the output for both is:

[u'\u4b03', u'\u98d2', u'[sa4]', u'/variant of \u98af|\u98d2[sa4]/']
[u'\u4b20', u'\u4b20', u'[fei1]', u'/old variant of \u970f[fei1]/']

My desired output:

䬃 飒 [sa4] /variant of 颯|飒[sa4]/
䬠 䬠 [fei1] /old variant of 霏[fei1]/

[EDIT BELOW]

lexicon.tsv contents:

䬃   飒   [sa4]   /variant of 颯|飒[sa4]/
䬠   䬠   [fei1]  /old variant of 霏[fei1]/

Answer 1

Apparently, I am receiving a list instead of a generator, so if in load_deck_data() I change:

 for field1, field2, field3, field4 in reader: print field1, field2, field3, field4 

to:

 for line in reader: print ''.join(line) 

my project works fine.

Check out this example:

data = [
    ['a', 'b', 'c', 'd'],
    ['e'],
]

def mygen(x):
    for item in x:
        yield item

for line in mygen(data):
    print ''.join(line)

--output:--
abcd
e

for col1, col2, col3, col4 in mygen(data):
    print col1, col2, col3, col4


--output:--
a b c d

Traceback (most recent call last):
  File "1.py", line 13, in <module>
    for col1, col2, col3, col4 in mygen(data):
ValueError: need more than 1 value to unpack

In the first for-in loop, you are asking, "Please retrieve all the elements in the list and join them together." In the second for-in loop, you are demanding, "Retrieve four elements from the list!" See the difference? In the first case, the list can contain 0 to n elements and there won't be an error. In the second case, the list has to have at least 4 elements--otherwise there will be an error.

I would love to know why I'm getting a generator in one place, but a list in another.

Simple. You aren't. csv.reader() returns a list of strings for each row, which means your generator function returns a list of strings for each iteration.

I think you changed the data in your file. In one file, you have tab delimited data, and csv.reader() returns a list of four things for each line in your file, which can be unpacked into four variables; but your other file has non-tab delimited data, which causes csv.reader() to read the whole line as one item, so the list of strings that csv.reader() returns contains only one item, and a one item list cannot be unpacked into four variables.

I tried just printing 'field1' in both cases. With that change the output for both is:

 [u'\䬃', u'\飒', u'[sa4]', u'/variant of \颯|\飒[sa4]/'] [u'\䬠', u'\䬠', u'[fei1]', u'/old variant of \霏[fei1]/'] 

Instead of doing print field1 , if you do print repr(field1) I suspect you will get:

"[u'\u4b03', u'\u98d2', u'[sa4]', u'/variant of \u98af|\u98d2[sa4]/']"

Note the outer quotes, which means your tsv file literally has the following on one line:

[䬃, 飒, [sa4], /variant of 颯|飒[sa4]/]

with no tabs separating anything, so the whole line-that-looks-like-a-list is read in as one item, therefore csv.reader() returns a list containing that one item. You were fooled into thinking the single item was a python list because when you print a string, python does not display the quotes. For example, there is no difference in the output for the following two print statements:

>>> print "[1, 2, 3]"
[1, 2, 3]
>>> print [1, 2, 3]
[1, 2, 3]

print can fool you in other situations as well because a string can contain unprintable characters, which the output of print won't reveal:

>>> print "hello\bworld"
hellworld

The bottom line is: you can never know what the original thing was by looking at the output of print. Whenever you want to know exactly what the original thing is, always use:

print repr(some_string)

Now, look at the results:

>>> print repr([1, 2, 3])
[1, 2, 3]
>>> print repr('[1, 2, 3]')
'[1, 2, 3]'
>>> print repr('hello\bworld')
'hello\x08world'

The output tells you exactly what the original thing was.

With the following tab delimited lexicon.tsv file:

1   2   3   €
䬃   飒   [sa4]   /variant of 颯|飒[sa4]/

the code below causes no errors after clicking on the Lexicon button:

from kivy.app import App
from kivy.uix.screenmanager import ScreenManager, Screen
import csv

class StudyScreenManager(ScreenManager):

    def unicode_csv_reader(self, utf8_data, dialect=csv.excel_tab, **kwargs):
        csv_reader = csv.reader(utf8_data, dialect=dialect, **kwargs)
        for row in csv_reader:
            yield [unicode(cell, 'utf-8') for cell in row]


    def load_deck_data(self):
        filename = 'lexicon.tsv'
        reader = self.unicode_csv_reader(open(filename))
        for field1, field2, field3, field4 in reader:
            print field1, field2, field3, field4


class HistoryScreen(Screen):
    pass

class MathScreen(Screen):
    pass

class MyApp(App):
    def build(self):
        sm = StudyScreenManager()
        sm.add_widget(HistoryScreen(name='history'))
        sm.add_widget(MathScreen(name='math'))

        return sm

MyApp().run()

my.kv:

<HistoryScreen>:  #the 'root' of the following widget hierarchy:
    BoxLayout:
        Button:
            text: 'Lexicon'
            on_press: app.root.load_deck_data()  #self=Button, root=HistoryScreen, app.root=the Widget returned by build()
        Button:
            text: "Next"
            on_press: root.manager.current = "math"

<MathScreen>: #the 'root' of the following widget heirarchy:
    BoxLayout:
        Button:
            text: 'Lexicon'
            on_press: app.root.load_deck_data()
        Button:
            text: 'Previous'
            on_press: root.manager.current = "history"

After clicking on the Lexicon button, here is the output I see in my utf-8 aware terminal window :

1 2 3 €
䬃 飒 [sa4] /variant of 颯|飒[sa4]/

Answer 2

Apparently, I am receiving a list instead of a generator, so if in load_deck_data() I change...:

for field1, field2, field3, field4 in reader:
    print field1, field2, field3, field4

...to...:

for line in reader:
    print ''.join(line)

...my project works fine. This, of course, does not work in the small code snippet that originally worked.

I would love to know why I'm getting a generator in one place, but a list in another. :)