如何在使用熊猫导入csv文件的过程中有效删除重叠的行？

Question

I am trying to import csv files with pandas that look basically like this:

File 1:

Date;Time;Value
2019-03-07;20:43;0.051
2019-03-07;20:44;0.048
...
2019-03-07;22:55;0.095
2019-03-07;22:56;0.098  

File 2:

Date;Time;Value
2019-03-07;22:55;0.095
2019-03-07;22:56;0.098    
...
2019-03-08;09:10;0.024
2019-03-08;09:11;0.022

Currently I am importing the data like this:

data = pd.concat([pd.read_csv(open(file),sep=';') for file in files])
data.index = pd.to_datetime(data['Date'] + ' ' + data['Time'])   

Obviously now I have the overlapping parts of the measurement data twice in my imported data frame, which plotted looks like this:

As I need to evaluate a large number of csv files I am interested what the most efficient way to handle a situation like this is.

I thought of these two options:

1. Import the files inside a loop and for each file only use the parts where file[i] > file[i-1] .
2. Import the files as I do right now and remove the duplicates in an additional step.

Which of these options is more efficient and is there maybe a more efficient option that I didn't think of right now?

Answer 1

As for removing duplicates, pandas has support for this:

data = pd.concat([pd.read_csv(open(file),sep=';') for file in files])
data.index = pd.to_datetime(data['Date'] + ' ' + data['Time']
data = data[~data.index.duplicated()]

See also docs: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Index.duplicated.html

Regarding the "best" way to do it, that depends on amount of data, other constraints, etc. Impossible to answer without more context and would likely be opinion based anyway.