将数据帧与熊猫合并

Question

I have multiple files containing dates and measured values. Their setup is identical:

YYYY  MM  DD  val1
YYYY  MM  DD  val2
YYYY  MM  DD  val3

I use the following to read each of these files into a DataFrame

for cur_file in file_list:
    cur_df = pa.io.parsers.read_table(os.path.join(data_path, result)
                                                , header=None
                                                , sep='\s*'
                                                , parse_dates=[[0,1, 2]]
                                                , names=['day','month', 'hour', cur_file[:-4]]
                                                , index_col=[0]
                                                )

The dates are not identical in all files. There is sometimes some overlap, but not always.

I could plot each of the cur_df individually via

cur_df.plot()

in the loop.

It seems like it would be a good idea to have all the cur_df in one "big" DataFrame. Both for plotting and also for statistics later on. How would this be done ideally, considering they have not the same dates? Is there a way to "merge" multiple DataFrames dates that occur only in one of the underlying DataFrames?

I guess I am looking for a data frame that looks like this:

YYYY MM DD  val1(from1)  NaN
YYYY MM DD  val2(from1)  val2(from2)
YYYY MM DD  NaN          val3(from2)

It would take the date stamp in the first line from the date of val1, in line two the dates of val1 and val2 are identical, and it would take the date in line 3 based on val2

I looked into cur_df.add(cur_df2) appends the two DataFrames. I am not sure what cur_df.combine(cur_df2, ...) would do, especially since I am not sure what function should be used as second argument.

Thanks for your help, Cheers, Claus

Answer 1

from your code snippet it looks like the parsed date value should be the index and each DataFrame will have the values in a different column name right? In that case I think an iterative call to DataFrame.combine_first should do the trick.

Also, are you passing in "keep_date_col=True" as well? By default the parser should be throwing away the component date columns when parsing multiple date components into one (if not then that's a bug so please let me know).

Best,

Chang