删除 pandas 时间序列中的重复项

Question

I have a csv file with a time series that has the structure: col1: date col2: value . The csv file has a date from, say, Jan 1st to April 30. I then have a second csv file with the difference that the date is Feb 1st until May 31. The values in the second column from February 1st until April 30 are the same in the first and second file. The same problem for a third csv file (March 1st until June 30), fourth etc.: same overalpping structure. I want to read these csv files but retain only unique dates from Jan 1st until, say, December 31 without repetitions in values. Is there a fast way to do this with Pandas dataframes?

Answer 1

One option is concat the files using pandas pd.concat() and then try:

df = pd.concat([file1,file2,file3])
df.drop_duplicates()

Answer 2

Without more info on your data, I'd probably do something like this:

df1, df2, df2 = load_your_data()  # pd.DataFrame objects

import pandas as pd
concat = pd.concat([df1, df2, df2], axis=0)
dedup = concat.drop_duplicates(subset=['col1'])

This assumes that your repeated dates are indeed duplicates, and you aren't losing any information by dropping these rows. Otherwise, I'd consider converting the dates to a DatetimeIndex , and resampling the data with an appropriate aggregation method.