Combining multiple data-frame columns

Question

I am trying to combine 2 data frames columns into 1 but when I try to do it based on specific size the second data-frame column doesn't copy correctly.

I have tried the code below as pasted below.

import pandas as pd
def readDataFile():
    fileName = "year.csv"
    dateparse = lambda x: pd.datetime.strptime(x, '%Y-%m-%d %H:%M:%S')
    dfY = pd.read_csv(fileName, parse_dates=['date'], date_parser=dateparse)

    fileName = "month.csv"
    dateparse = lambda x: pd.datetime.strptime(x, '%Y-%m-%d %H:%M:%S')
    dfM = pd.read_csv(fileName, parse_dates=['date'], date_parser=dateparse)


    newDF = pd.DataFrame()
    newDF['date_y'] = dfY['date']
    newDF['year_y_n'] = dfY['Y_N']
    newDF['date_m'] = dfM['date'][len(dfM) - len(dfY):len(dfM)]
    newDF['year_y_n'] = dfM['Y_N'][len(dfM) - len(dfY):len(dfM)]
    print newDF
readDataFile()

File: month.csv

date,Y_N
2018-03-14 04:00:00,N
2018-04-03 04:00:00,N
2018-05-31 04:00:00,Y
2018-06-14 04:00:00,N
2018-07-30 04:00:00,N
2018-08-31 04:00:00,Y
2018-09-28 04:00:00,N
2018-10-10 04:00:00,N
2018-11-07 04:00:00,Y
2018-12-31 04:00:00,N
2019-01-31 04:00:00,N
2019-02-05 04:00:00,Y
2019-03-29 04:00:00,N
2019-04-30 04:00:00,Y
2019-05-03 04:00:00,N
2019-06-03 04:00:00,Y

File: year.csv

date,Y_N
2014-05-23 04:00:00,Y
2015-12-21 04:00:00,N
2016-05-03 04:00:00,Y
2017-12-20 04:00:00,N
2018-06-14 04:00:00,N
2019-06-25 04:00:00,N

These are the CURRENT results:

date_y year_y_n date_m month_y_n
0 2014-05-23 04:00:00        Y    NaT       NaN
1 2015-12-21 04:00:00        N    NaT       NaN
2 2016-05-03 04:00:00        Y    NaT       NaN
3 2017-12-20 04:00:00        N    NaT       NaN
4 2018-06-14 04:00:00        N    NaT       NaN
5 2019-06-25 04:00:00        N    NaT       NaN

Expected results are:

date_y              year_y_n    date_m              month_y_n
2014-05-23 04:00:00        Y  2019-01-31 04:00:00       N
2015-12-21 04:00:00        N  2019-02-05 04:00:00       Y
2016-05-03 04:00:00        Y  2019-03-29 04:00:00       N
2017-12-20 04:00:00        N  2019-04-30 04:00:00       Y
2018-06-14 04:00:00        N  2019-05-03 04:00:00       N
2019-06-25 04:00:00        N  2019-06-03 04:00:00       Y

Answer 1

Let's say you have an arbitrary number of dataframes dfA , dfB , dfC , etc. You want to merge them, but they're different sizes. The most basic approach is to concatenate them:

df = pd.concat([dfA, dfB, dfC], axis=1)

But if the dataframes are different sizes, there will be missing rows. If you don't care which rows are preserved, you can just drop rows with missing values:

df.dropna()

But if you specifically want to use the last N rows of each dataframe, where N is the length of the smallest dataframe, you need to do a little more work. But I'll wait and see if that's what you want.

---

Old Answer:

Merging can be much simpler than this. Use pd.merge :

pd.merge(dfY, dfM[-len(dfY):].reset_index(), 
    suffixes=['_y', '_m'], left_index=True, right_index=True)

• dfM[-len(dfY):] gets the last N rows of dfM , where N is the length of dfY .
• .reset_index() makes the index of the subset of dfM start at 0, so it can correctly align with dfY .
• suffixes=['_y', '_m'] keeps the column names different. You can rename these if you like.

Answer 2

The problem was related to the index. If you run the code below:

newDF = pd.DataFrame()
newDF['date_y'] = dfY['date']
print(newDF)

You will get the output:

     date_y
0 2014-05-23 04:00:00
1 2015-12-21 04:00:00
2 2016-05-03 04:00:00
3 2017-12-20 04:00:00
4 2018-06-14 04:00:00
5 2019-06-25 04:00:00

The index is starting from 0

And running this:

newDF = pd.DataFrame()
newDF['date_m'] = dfM['date'][len(dfM) - len(dfY):len(dfM)]
print(newDF)

You will get the output:

    date_m
10 2019-01-31 04:00:00
11 2019-02-05 04:00:00
12 2019-03-29 04:00:00
13 2019-04-30 04:00:00
14 2019-05-03 04:00:00
15 2019-06-03 04:00:00

Here, the index is starting from 10

So, you need to reset the index of the columns 'date' and 'Y_N' of dfM dataframe,like this:

def readDataFile():
    fileName = "year.csv"
    dateparse = lambda x: pd.datetime.strptime(x, '%Y-%m-%d %H:%M:%S')
    dfY = pd.read_csv(fileName, parse_dates=['date'], date_parser=dateparse)

    fileName = "month.csv"
    dateparse = lambda x: pd.datetime.strptime(x, '%Y-%m-%d %H:%M:%S')
    dfM = pd.read_csv(fileName, parse_dates=['date'], date_parser=dateparse)


    newDF = pd.DataFrame()
    newDF['date_y'] = dfY['date']
    newDF['year_y_n'] = dfY['Y_N']

    # Changes made on this line.
    newDF['date_m'] = dfM['date'][len(dfM) - len(dfY):len(dfM)].reset_index(drop=True)
    newDF['month_y_n'] = dfM['Y_N'][len(dfM) - len(dfY):len(dfM)].reset_index(drop=True)

    print(newDF)
readDataFile()

Output:

date_y year_y_n              date_m month_y_n
0 2014-05-23 04:00:00        Y 2019-01-31 04:00:00         N
1 2015-12-21 04:00:00        N 2019-02-05 04:00:00         Y
2 2016-05-03 04:00:00        Y 2019-03-29 04:00:00         N
3 2017-12-20 04:00:00        N 2019-04-30 04:00:00         Y
4 2018-06-14 04:00:00        N 2019-05-03 04:00:00         N
5 2019-06-25 04:00:00        N 2019-06-03 04:00:00         Y