遍历字典，一次 5 行

Question

I am trying to open a csv file with csv.DictReader, read in just the first 5 rows of data, perform the primary process of my script, then read in the next 5 rows and do the same for them. Rinse and repeat.

I believe I have a method that works, however I am having issues with the last lines of the data not processing. I know I need to modify my if statement so that it also checks for if I'm at the end of the file, but am having trouble finding a way to do that. I've found methods online, but they involve reading in the whole file to get a row count but doing so would defeat the purpose of this script as I'm dealing with memory issues.

Here is what I have so far:

import csv
count = 0
data = []
with open('test.csv') as file:
    reader = csv.DictReader(file)
    
    for row in reader:
        count +=1
        data.append(row)

        if count % 5 == 0 or #something to check for the end of the file:
            #do stuff
            data = []
        

Thank you for the help!

Answer 1

You can use the chunksize argument when reading in the csv. This will step by step read in the number of lines:

import pandas as pd
reader = pd.read_csv('test.csv', chunksize=5)
for df in reader:
    # do stuff

Answer 2

You can handle the remaining lines after the for loop body. You can also use the more pythonic enumerate .

import csv

data = []
with open('test.csv') as file:
    reader = csv.DictReader(file)
    for count, row in enumerate(reader, 1):
        data.append(row)
        if count % 5 == 0:
            # do stuff
            data = []

    print('handling remaining lines at end of file')
    print(data)

considering the file

a,b
1,1
2,2
3,3
4,4
5,5
6,6
7,7

outputs

handling remaining lines at end of file
[OrderedDict([('a', '6'), ('b', '6')]), OrderedDict([('a', '7'), ('b', '7')])]

Answer 3

This is one approach using the iterator

Ex:

import csv

with open('test.csv') as file:
    reader = csv.DictReader(file)
    
    value = True
    while value:
        data = []
        for _ in range(5):             # Get 5 rows
            value = next(reader, False) 
            if value:
                data.append(value)
        print(data)   #List of 5 elements

Answer 4

Staying along the lines of what you wrote and not including any other imports:

import csv
data = []
with open('test.csv') as file:
    reader = csv.DictReader(file)

    for row in reader:
        data.append(row)
        if len(data) > 5:
            del data[0]
        if len(data) == 5:
            # Do something with the 5 elements
            print(data)

The if statements allow the array to be loaded with 5 elements before processing on the begins.

class ZeroItterNumberException(Exception):
    pass
class ItterN:
    def __init__(self, itterator, n):
        if n<1:
            raise ZeroItterNumberException("{} is not a valid number of rows.".format(n))
        self.itterator = itterator
        self.n = n
        self.cache = []

    def __iter__(self):
        return self

    def __next__(self):
        self.cache.append(next(self.itterator))
        if len(self.cache) < self.n:
            return self.__next__()
        if len(self.cache) > self.n:
            del self.cache[0]
        if len(self.cache) == 5:
            return self.cache