[英]Optimum way to transpose rows in one dataframe to columns in another dataframe in Pandas?
Given df_people :鉴于df_people :
Name
0 Tom
1 Jerry
and df_colors (no header row):和df_colors (无标题行):
0 Red
1 Green
2 Blue
What is considered an optimum way to take the data in df_colors and add it to df_people such that df_people would look like this when combined:什么被认为是采取df_colors的数据,并把它添加到df_people这样df_people看起来像这样结合时的最佳方式:
Name Color_0 Color_1 Color_2
0 Tom Red Green Blue
1 Jerry Red Green Blue
Below is what I have so far, which works, but I was wondering if there is a better or more concise way.以下是我到目前为止所拥有的,它有效,但我想知道是否有更好或更简洁的方法。
# Store data for new columns in a dictionary
new_columns = {}
for index_people, row_people in df_people.iterrows():
for index_colors, row_colors in df_colors.iterrows():
key = 'Color_' + str(index_colors)
if (key in new_columns):
new_columns[key].append(row_colors[0])
else:
new_columns[key] = [row_colors[0]]
# Add dictionary data as new columns
for key, value in new_columns.items():
df_people[key] = value
UPDATE更新
Thank you all for providing answers.谢谢大家提供答案。 Since the real dataframes are GBs in size, speed was crucial, so I ended up going with the fastest method.
由于实际数据帧的大小为 GB,因此速度至关重要,因此我最终采用了最快的方法。 Here is the code to the test cases:
下面是测试用例的代码:
# Import required modules
import pandas as pd
import timeit
# Original
def method_1():
df_people = pd.DataFrame([['Tom'], ['Jerry']], columns=['Name'])
df_colors = pd.DataFrame([['Red'], ['Green'], ['Blue']], columns=None)
# Store data for new columns in a dictionary
new_columns = {}
for index_people, row_people in df_people.iterrows():
for index_colors, row_colors in df_colors.iterrows():
key = 'Color_' + str(index_colors)
if (key in new_columns):
new_columns[key].append(row_colors[0])
else:
new_columns[key] = [row_colors[0]]
# Add dictionary data as new columns
for key, value in new_columns.items():
df_people[key] = value
# YOBEN_S - https://stackoverflow.com/a/60805881/452587
def method_2():
df_people = pd.DataFrame([['Tom'], ['Jerry']], columns=['Name'])
df_colors = pd.DataFrame([['Red'], ['Green'], ['Blue']], columns=None)
_s = pd.concat([df_colors]*len(df_people), axis=1)
_s.columns = df_people.index
df_people = df_people.join(_s.T.add_prefix('Color_'))
# Dani Mesejo - https://stackoverflow.com/a/60805898/452587
def method_3():
df_people = pd.DataFrame([['Tom'], ['Jerry']], columns=['Name'])
df_colors = pd.DataFrame([['Red'], ['Green'], ['Blue']], columns=None)
# Create mock key
_m1 = df_people.assign(key=1)
# Set new column names, transpose, and create mock key
_m2 = df_colors.set_index('Color_' + df_colors.index.astype(str)).T.assign(key=1)
df_people = _m1.merge(_m2, on='key').drop('key', axis=1)
# Erfan - https://stackoverflow.com/a/60806018/452587
def method_4():
df_people = pd.DataFrame([['Tom'], ['Jerry']], columns=['Name'])
df_colors = pd.DataFrame([['Red'], ['Green'], ['Blue']], columns=None)
df_colors = df_colors.T.reindex(df_people.index).ffill().add_prefix('Color_')
df_people = df_people.join(df_colors)
print('Method 1:', timeit.timeit(method_1, number=10000))
print('Method 2:', timeit.timeit(method_2, number=10000))
print('Method 3:', timeit.timeit(method_3, number=10000))
print('Method 4:', timeit.timeit(method_4, number=10000))
Output:输出:
Method 1: 36.029883089
Method 2: 27.042384837999997
Method 3: 68.22421793800001
Method 4: 32.94155895
In my effort to simplify the scenario, unfortunately I oversimplified it.在我努力简化场景的过程中,不幸的是我过于简化了它。 It's too late to rephrase the question now, so I think I will post a related question at a later date.
现在改写这个问题已经太晚了,所以我想我会在以后发布一个相关的问题。 The real scenario involves mathematics as well, so instead of simply appending columns in
df_colors
to df_people
, I also need to perform some calculations against a column in the corresponding row for each added cell.实际场景也涉及数学,因此不是简单地将
df_colors
列附加到df_people
,我还需要对每个添加的单元格的相应行中的列执行一些计算。
UPDATE 2更新 2
I've made the sample dataframes larger (thanks jezrael) and added two new methods.我已经使示例数据帧更大(感谢 jezrael)并添加了两种新方法。
# Import required modules
import numpy as np
import pandas as pd
import timeit
# Original
def method_1():
df_people = pd.DataFrame(['Tom', 'Jerry', 'Bob', 'John', 'Bill', 'Tim', 'Harry', 'Rick'] * 1000, columns=['Name'])
df_colors = pd.DataFrame(['Red', 'Green', 'Blue'] * 10, columns=None)
# Store data for new columns in a dictionary
new_columns = {}
for index_people, row_people in df_people.iterrows():
for index_colors, row_colors in df_colors.iterrows():
key = 'Color_' + str(index_colors)
if (key in new_columns):
new_columns[key].append(row_colors[0])
else:
new_columns[key] = [row_colors[0]]
# Add dictionary data as new columns
for key, value in new_columns.items():
df_people[key] = value
# YOBEN_S - https://stackoverflow.com/a/60805881/452587
def method_2():
df_people = pd.DataFrame(['Tom', 'Jerry', 'Bob', 'John', 'Bill', 'Tim', 'Harry', 'Rick'] * 1000, columns=['Name'])
df_colors = pd.DataFrame(['Red', 'Green', 'Blue'] * 10, columns=None)
_s = pd.concat([df_colors]*len(df_people), axis=1)
_s.columns = df_people.index
df_people = df_people.join(_s.T.add_prefix('Color_'))
# sammywemmy - https://stackoverflow.com/a/60805964/452587
def method_3():
df_people = pd.DataFrame(['Tom', 'Jerry', 'Bob', 'John', 'Bill', 'Tim', 'Harry', 'Rick'] * 1000, columns=['Name'])
df_colors = pd.DataFrame(['Red', 'Green', 'Blue'] * 10, columns=None)
# Create a new column in df_people with aggregate of df_colors;
df_people['Colors'] = df_colors[0].str.cat(sep=',')
# Concatenate df_people['Name'] and df_people['Colors'];
# split column, expand into a dataframe, and add prefix
df_people = pd.concat([df_people.Name, df_people.Colors.str.split(',', expand=True).add_prefix('Color_')], axis=1)
# Dani Mesejo - https://stackoverflow.com/a/60805898/452587
def method_4():
df_people = pd.DataFrame(['Tom', 'Jerry', 'Bob', 'John', 'Bill', 'Tim', 'Harry', 'Rick'] * 1000, columns=['Name'])
df_colors = pd.DataFrame(['Red', 'Green', 'Blue'] * 10, columns=None)
# Create mock key
_m1 = df_people.assign(key=1)
# Set new column names, transpose, and create mock key
_m2 = df_colors.set_index('Color_' + df_colors.index.astype(str)).T.assign(key=1)
df_people = _m1.merge(_m2, on='key').drop('key', axis=1)
# Erfan - https://stackoverflow.com/a/60806018/452587
def method_5():
df_people = pd.DataFrame(['Tom', 'Jerry', 'Bob', 'John', 'Bill', 'Tim', 'Harry', 'Rick'] * 1000, columns=['Name'])
df_colors = pd.DataFrame(['Red', 'Green', 'Blue'] * 10, columns=None)
df_colors = df_colors.T.reindex(df_people.index).ffill().add_prefix('Color_')
df_people = df_people.join(df_colors)
# jezrael - https://stackoverflow.com/a/60826723/452587
def method_6():
df_people = pd.DataFrame(['Tom', 'Jerry', 'Bob', 'John', 'Bill', 'Tim', 'Harry', 'Rick'] * 1000, columns=['Name'])
df_colors = pd.DataFrame(['Red', 'Green', 'Blue'] * 10, columns=None)
_a = np.broadcast_to(df_colors[0], (len(df_people), len(df_colors)))
df_people = df_people.join(pd.DataFrame(_a, index=df_people.index).add_prefix('Color_'))
print('Method 1:', timeit.timeit(method_1, number=3))
print('Method 2:', timeit.timeit(method_2, number=3))
print('Method 3:', timeit.timeit(method_3, number=3))
print('Method 4:', timeit.timeit(method_4, number=3))
print('Method 5:', timeit.timeit(method_5, number=3))
print('Method 6:', timeit.timeit(method_6, number=3))
Output:输出:
Method 1: 74.512771493
Method 2: 1.0007798979999905
Method 3: 0.40823360299999933
Method 4: 0.08115736700000298
Method 5: 0.11704620100000795
Method 6: 0.04700596800000767
UPDATE 3更新 3
I've posted a related question for transposing and calculating, which more accurately reflects the real dataset:我已经发布了一个有关转置和计算的相关问题,它更准确地反映了真实数据集:
Fastest way to transpose and calculate in Pandas?在 Pandas 中转置和计算的最快方法?
We can do我们可以做的
s=pd.concat([df1]*len(df),axis=1)
s.columns=df.index
df=df.join(s.T.add_prefix('color_'))
Name color_0 color_1 color_2
0 Tom Red Green Blue
1 Jerry Red Green Blue
You can improve performance by numpy.broadcast_to
method:您可以通过
numpy.broadcast_to
方法提高性能:
df_people = pd.DataFrame([['Tom'], ['Jerry']], columns=['Name'])
df_colors = pd.DataFrame([['Red'], ['Green'], ['Blue']], columns=None)
a = np.broadcast_to(df_colors[0], (len(df_people), len(df_colors)))
df = df_people.join(pd.DataFrame(a, index=df_people.index).add_prefix('Color_'))
print (df)
Name Color_0 Color_1 Color_2
0 Tom Red Green Blue
1 Jerry Red Green Blue
import timeit
def method_2():
df_people = pd.DataFrame([['Tom'], ['Jerry']], columns=['Name'])
df_colors = pd.DataFrame([['Red'], ['Green'], ['Blue']], columns=None)
_s = pd.concat([df_colors]*len(df_people), axis=1)
_s.columns = df_people.index
df_people = df_people.join(_s.T.add_prefix('Color_'))
def method_5():
df_people = pd.DataFrame([['Tom'], ['Jerry']], columns=['Name'])
df_colors = pd.DataFrame([['Red'], ['Green'], ['Blue']], columns=None)
a = np.broadcast_to(df_colors[0], (len(df_people), len(df_colors)))
df_people = df_people.join(pd.DataFrame(a, index=df_people.index).add_prefix('Color_'))
print('Method 2:', timeit.timeit(method_2, number=10000))
Method 2: 27.919169027998578
print('Method 5:', timeit.timeit(method_5, number=10000))
Method 5: 21.452649746001043
But I think better is test in large DataFrame
, eg here for 3k rows and 30 columns, then timings are different:但我认为更好的是在大型
DataFrame
测试,例如这里有 3k 行和 30 列,然后时间不同:
# Import required modules
import pandas as pd
import timeit
# Original
def method_1():
df_people = pd.DataFrame(['Tom','Jerry','Bob'] * 1000, columns=['Name'])
df_colors = pd.DataFrame(['Red','Green', 'Blue'] * 10, columns=None)
# Store data for new columns in a dictionary
new_columns = {}
for index_people, row_people in df_people.iterrows():
for index_colors, row_colors in df_colors.iterrows():
key = 'Color_' + str(index_colors)
if (key in new_columns):
new_columns[key].append(row_colors[0])
else:
new_columns[key] = [row_colors[0]]
# Add dictionary data as new columns
for key, value in new_columns.items():
df_people[key] = value
# YOBEN_S - https://stackoverflow.com/a/60805881/452587
def method_2():
df_people = pd.DataFrame(['Tom','Jerry','Bob'] * 1000, columns=['Name'])
df_colors = pd.DataFrame(['Red','Green', 'Blue'] * 10, columns=None)
_s = pd.concat([df_colors]*len(df_people), axis=1)
_s.columns = df_people.index
df_people = df_people.join(_s.T.add_prefix('Color_'))
# Dani Mesejo - https://stackoverflow.com/a/60805898/452587
def method_3():
df_people = pd.DataFrame(['Tom','Jerry','Bob'] * 1000, columns=['Name'])
df_colors = pd.DataFrame(['Red','Green', 'Blue'] * 10, columns=None)
# Create mock key
_m1 = df_people.assign(key=1)
# Set new column names, transpose, and create mock key
_m2 = df_colors.set_index('Color_' + df_colors.index.astype(str)).T.assign(key=1)
df_people = _m1.merge(_m2, on='key').drop('key', axis=1)
# Erfan - https://stackoverflow.com/a/60806018/452587
def method_4():
df_people = pd.DataFrame(['Tom','Jerry','Bob'] * 1000, columns=['Name'])
df_colors = pd.DataFrame(['Red','Green', 'Blue'] * 10, columns=None)
df_colors = df_colors.T.reindex(df_people.index).ffill().add_prefix('Color_')
df_people = df_people.join(df_colors)
def method_5():
df_people = pd.DataFrame(['Tom','Jerry','Bob'] * 1000, columns=['Name'])
df_colors = pd.DataFrame(['Red','Green', 'Blue'] * 10, columns=None)
a = np.broadcast_to(df_colors[0], (len(df_people), len(df_colors)))
df_people = df_people.join(pd.DataFrame(a, index=df_people.index).add_prefix('Color_'))
print('Method 1:', timeit.timeit(method_1, number=3))
print('Method 2:', timeit.timeit(method_2, number=3))
print('Method 3:', timeit.timeit(method_3, number=3))
print('Method 4:', timeit.timeit(method_4, number=3))
print('Method 5:', timeit.timeit(method_5, number=3))
Method 1: 34.91457201199955
Method 2: 0.7901797180002177
Method 3: 0.05690281799979857
Method 4: 0.05774562500118918
Method 5: 0.026483284000278218
You could do:你可以这样做:
import pandas as pd
# input sample data
df1 = pd.DataFrame([['Tom'], ['Jerry']], columns=['name'])
df2 = pd.DataFrame([['Red'], ['Gree'], ['Blue']], columns=None)
# create mock key
m1 = df1.assign(key=1)
# set new column names, transpose and create mock key
m2 = df2.set_index('Color_' + df2.index.astype(str)).T.assign(key=1)
result = m1.merge(m2, on='key').drop('key', axis=1)
print(result)
Output输出
name Color_0 Color_1 Color_2
0 Tom Red Gree Blue
1 Jerry Red Gree Blue
Another possible solution:另一种可能的解决方案:
#create a new column in df1, with aggregate of df2:
#i set the header for df2 column as 'color'
df1['color'] = df2['color'].str.cat(sep=',')
#concatenate df1['Name'] and df1['Color'] as below:
pd.concat([df1.Name,
#split column, expand into a dataframe and add prefix
df1.color.str.split(',',expand=True).add_prefix('color_')],
axis=1)
Name color_0 color_1 color_2
0 Tom Red Green Blue
1 Jerry Red Green Blue
Using DataFrame.reindex
, DataFrame.ffill
and DataFrame.add_prefix
:使用
DataFrame.reindex
、 DataFrame.ffill
和DataFrame.add_prefix
:
df2 = df2.T.reindex(df1.index).ffill().add_prefix('Color_')
df1 = df1.join(df2)
Name Color_0 Color_1 Color_2
0 Tom Red Green Blue
1 Jerry Red Green Blue
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.