使用pd.read_csv时跳过包含错误日期的行

Question

I'm reading in csv files from an external data source using pd.read_csv , as in the code below:

pd.read_csv(
    BytesIO(raw_data),
    parse_dates=['dates'],
    date_parser=np.datetime64,
)

However, somewhere in the csv that's being sent, there is a misformatted date, resulting in the following error:

ValueError: Error parsing datetime string "2015-08-2" at position 8

This causes the entire application to crash. Of course, I can handle this case with a try/except, but then I will lose all the other data in that particular csv. I need pandas to keep and parse that other data.

I have no way of predicting when/where this data (which changes daily) will have badly formatted dates. Is there some way to get pd.read_csv to skip only the rows with bad dates but to still parse all the other rows in the csv?

Answer 1

somewhere in the csv that's being sent, there is a misformatted date

np.datetime64 needs ISO8601 formatted strings to work properly. The good news is that you can wrap np.datetime64 in your own function and use this as the date_parser :

def parse_date(v):
   try:
      return np.datetime64(v)
   except:
      # apply whatever remedies you deem appropriate
      pass
   return v

   pd.read_csv(
     ...
     date_parser=parse_date
   )

I need pandas to keep and parse that other data.

I often find that a more flexible date parser like dateutil works better than np.datetime64 and may even work without the extra function:

import dateutil
pd.read_csv(
    BytesIO(raw_data),
    parse_dates=['dates'],
    date_parser=dateutil.parser.parse,
)

Answer 2

Here's another way to do this using pd.convert_objects() method:

# make good and bad date csv files
# read in good dates file using parse_dates - no problem
df = pd.read_csv('dategood.csv', parse_dates=['dates'], date_parser=np.datetime64)

df.dtypes

dates    datetime64[ns]
data            float64
dtype: object

# try same code on bad dates file - throws exceptions
df = pd.read_csv('datebad.csv', parse_dates=['dates'], date_parser=np.datetime64)

ValueError: Error parsing datetime string "Q%Bte0tvk5" at position 0

# read the file first without converting dates
# then use convert objects to force conversion
df = pd.read_csv('datebad.csv')
df['cdate'] = df.dates.convert_objects(convert_dates='coerce')

# resulting new date column is a datetime64 same as good data file
df.dtype

dates            object
data            float64
cdate    datetime64[ns]
dtype: object

# the bad date has NaT in the cdate column - can clean it later
df.head()

        dates      data      cdate
0  2015-12-01  0.914836 2015-12-01
1  2015-12-02  0.866848 2015-12-02
2  2015-12-03  0.103718 2015-12-03
3  2015-12-04  0.514086 2015-12-04
4  Q%Bte0tvk5  0.583617        NaT

Answer 3

use inbuilt pd.to_datetime , which converts the non date type data to NaT

pd.read_csv(
    BytesIO(raw_data),
    parse_dates=['dates'],
    date_parser=pd.to_datetime,
)

Now you can filter out the invalid rows with standard nan/ null check

df = df[~df["dates"].isnull()]