I have a collection of files that have some common columns that I want to join. In my real problem, there are several dissimilar and common columns. In this toy example, I have a set of a
files and a set of b
files that have unique columns and share identical c
columns.
$ for ii in $(ls *.dat) ; do echo " "; echo $ii ; cat $ii ; done
a1.dat
a,c
4,8
1,10
2,3
a2.dat
a,c
1,2
3,4
b1.dat
b,c
2,8
2,10
1,3
b2.dat
b,c
.2,2
.8,4
I want to sweep through these files and merge them into a single dataframe. Here's what I've tried so far. I concat the first files to make sure I have all of the column names collected then merge the remaining files. When I merge by "inner", an empty dataframe is returned.
$ cat s.py
import pandas as pd
dat = pd.DataFrame()
for ii in [1, 2]:
for jj in ['a', 'b']:
d = pd.read_csv('%s%i.dat' % (jj, ii))
if ii == 1: dat = pd.concat([dat, d])
else: dat = pd.merge(dat, d, how='outer')
print(dat)
$ Python s.py
a b c
0 4.0 NaN 8
1 1.0 NaN 10
2 2.0 NaN 3
3 NaN 2.0 8
4 NaN 2.0 10
5 NaN 1.0 3
6 1.0 NaN 2
7 3.0 NaN 4
8 NaN 0.2 2
9 NaN 0.8 4
This is not my desired output. I don't understand how I can make this work better. The desired output was
a b c
0 4.0 2.0 8
1 1.0 2.0 10
2 2.0 1.0 3
3 1.0 0.2 2
4 3.0 0.8 4
There are two steps:
First, concatenate all files of the same type into one DataFrame each:
df = {}
for k in ['a', 'b']:
df[k] = pd.concat([
pd.read_csv('%s%d.dat' % (k, i)) for i in [1, 2]
], axis=0)
Then merge join on the shared column 'c',
result = df['a'].merge(df['b'], on='c')[['a', 'b', 'c']]
First concat all a and b files and then merge them on the column c like:
import numpy as np
import pandas as pd
a1 = pd.DataFrame({
'a': [4,1,2],
'c': [8,10,3],
})
a2 = pd.DataFrame({
'a': [1,3],
'c': [2,4],
})
b1 = pd.DataFrame({
'b': [2,2,1],
'c': [8,10,3],
})
b2 = pd.DataFrame({
'b': [0.2,0.8],
'c': [2,4],
})
concat_df_a = pd.concat([a1,a2])
concat_df_b = pd.concat([b1,b2])
print(concat_df_b.merge(concat_df_a,on='c')[['a','b','c']])
a b c
0 4 2.0 8
1 1 2.0 10
2 2 1.0 3
3 1 0.2 2
4 3 0.8 4
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.