遍历 csv 文件

Question

Hi I am trying to iterate through a csv file but I cannot get it to work somehow. I followed the python docs but I am still not able to iterate through it. I have a gzipped csv file that I work with with this format:

2015-01-10 00:00:05;32

As you can see it's delimited with a ';'.

Here is my code to run though it (simplified)

 gzip_fd = gzip.decompress(gzip_file).decode(encoding='utf8')
 csv_data = csv.reader(gzip_fd, delimiter=';', lineterminator='\n')
 for data in csv_data:
     print(data)

But when I want to work with data it only contains the first character (like: 2) and not the first part of the csv data that I need. Anyone here that had the same issues? I also tried csv.DictReader but with no success.

Answer 1

Even if your snippet was fixed to work, it would buffer all data in the memory, which might not scale well for very large files.

Gzipped data can also be iterated on-the-fly -- the following works for me on CPython 3.8:

import csv
import gzip


with gzip.open('test.csv.gz', 'r') as gzipped:
    reader = csv.reader(gzipped, delimiter=';', lineterminator='\n')

    for line in reader:
        print(line)

['2015-01-10 00:00:05', '32']

<...>

Update : As per comments below, my snippet does not work on older Python versions (reproduced on CPython 3.5).

You can use io.TextIOWrapper to achieve the same effect:

import csv
import io
import gzip


with gzip.open('test.csv.gz', 'rb') as gzipped:
    reader = csv.reader(io.TextIOWrapper(gzipped), delimiter=';',
                        lineterminator='\n')

    for line in reader:
        print(line)

Answer 2

So I fixed my issue, the issue was that I didn't split the string that I get (can't do gzip.open because it isn't a file but rather a bytes string of the gzipped file

Here is the fix to my problem:

gzip_fd = gzip.decompress(compressed_data).decode(encoding='utf-8').split('\n')
self.data = csv.reader(gzip_fd, delimiter=';', lineterminator='\n')