如何使用 Python (200 GB+) 从长 csv 文件的中间读取块

Question

I have a large csv file and I am reading it with chunks. In the middle of the process memory got full so I want to restart from where it left. I know which chunk but don't know how to go to that chunk directly.

This is what I tried.

# data is the txt file
reader = pd.read_csv(data , 
                     delimiter = "\t",
                     chunksize = 1000
                    )


# Please see the code below. When my last process broke, i was 154 so I think it should 
# start from 154000th line. This time I don't 
# plan to read whole file at once so I have an 
# end point at 160000

first = 154*1000
last = 160*1000

output_path = 'usa_hotspot_data_' + str(first) + '_' + str(last) + '.csv'
print("Output file: ", output_path)

try:
    os.remove(output_path)
except OSError:
    pass

# Read chunks and save to a new csv
for i,chunk in enumerate(reader):
    if (i >= first and i<=last) :
          < -- here I do something  -- > 
        # Progress Bar to keep track 
        if (i% 1000 == 0):
            print("#", end ='')

However, this is taking a lot of time to reach the ith line I want to go. How can I skip reading chunks before it and directly go there?

Answer 1

pandas.read_csv

skiprows : Line numbers to skip (0-indexed) or number of lines to skip (int) at the start of the file.

You can pass this skiprows to read_csv , It will act like offset.