如何使用 python 仅读取 a.csv 中特定范围的行？

Question

I am trying to process a huge.csv file but I don't need the first ~900000 rows of data. This is how I was originally trying to get rid of that chunk of data, but it makes the program take forever to finish. Is there a more straightforward way to do this where I don't even read those first 900000 rows in the first place?

firstColumn = [ ]
secondColumn = [ ]
thirdColumn = [ ]

readFile  = input("Enter name of file to be read: ")

with open(readFile,'r') as readFile:

    for eachline in readFile:                               # converting columns to lists
        parts = eachline.strip('\n').split(',')
        firstColumn.append(parts[0])
        secondColumn.append(parts[1])
        thirdColumn.append(parts[2])    
    
for j in range(900000):                          # nothing happens for these datapoints
    del firstColumn[j]
    del secondColumn[j]
    del thirdColumn[j]

Answer 1

You can skip the initial lines by doing something like this:

with open(readFile, 'r') as f:
    # skip first 900,000 lines
    for _ in range(900000):
        next(f)
    for line in f:
        parts = line.strip('\n').split(',')
        firstColumn.append(parts[0])
        secondColumn.append(parts[1])
        thirdColumn.append(parts[2])

Answer 2

You're right; that's awful . It's silly to convert 900K lines of input that you don't intend to use. Instead, skip past them entirely:

# read past first 900K lines
with open(readFile,'r') as readFile:
    for _ in range(900000):
        readFile.readline()

    for eachline in readFile:                               # converting columns to lists
        # Continue as before

With that done, I strongly recommend that you switch to a csv reader to grab the rest of the file; you can build your data frame in on simple operation from there. Be careful that you do not close and reopen the file, or otherwise reset the file header info.

Answer 3

You could use pandas which has a way to make a copy.csv file with those rows eliminated in the copy. First make a dataframe of your file, then you can use.iloc[] to put the row index in that you want to start from. This would be after the rows you want to cut. Parse it like a string.