删除 .csv 文件中的行 (Python)

Question

Good evening, I'm having a problem with a code I'm writing, and I would love to get advice. I want to do the following:

1. Remove rows in a .csv file that contain a specific value (-3.4028*10^38)
2. Write a new .csv

The file I'm working with is large (12.2 GB, 87 million rows), and has 6 columns within it, with the first 5 columns being numerical values, and the last value containing text.

Here is my code:

import csv

directory = "/media/gman/Folder1/processed/test_removal1.csv"
with open('run1.csv', 'r') as fin, open(directory, 'w', newline='') as fout:

# define reader and writer objects
reader = csv.reader(fin, skipinitialspace=False)
writer = csv.writer(fout, delimiter=',')

# write headers
writer.writerow(next(reader))

# iterate and write rows based on condition
for i in reader:
    if (i[-1]) == -3.4028E38:
        writer.writerow(i)

When I run this I get the following error message:

Error: line contains NUL

File "/media/gman/Aerospace_Classes/Programs/csv_remove.py", line 19, in <module>
for i in reader: Error: line contains NUL 

I'm not sure how to proceed. If anyone has any suggestions, please let me know. Thank you.

Answer 1

I figured it out. Here is what I ended up doing:

#IMPORT LIBRARIES
import pandas as pd

#IMPORT FILE PATH
directory = '/media/gman/Grant/Maps/processed_maps/csv_combined.csv'

#CREATE DATAFRAME FROM IMPORTED CSV
data = pd.read_csv(directory)
data.head()
data.drop(data[data.iloc[:,2] < -100000].index, inplace=True) #remove rows that contain altitude values greater than -100,000 meters.
# this is to remove the -3.402823E038 meter altitude values that keep coming up.

#CONVERT PROCESSED DATAFRAME INTO NEW CSV FILE
df = data.to_csv(r'/media/gman/Grant/Maps/processed_maps/corrected_altitude_data.csv') #export good data to this file.

I went with pandas to remove rows based on a logic argument, this made a dataframe. I then exported the dataframe into a csv file.