将来自不同文件夹的多个csv文件中的选定列合并到单个csv文件中

Question

This is for creating the final dataframe for my analysis.So I have three kinds of csv files.I want to extract specific columns from file 1 & file 2 and concatenate it to file 3 to get a single csv file.

I have one folder with the three types of files as subfolders-> that is 3 subfolders These 3 subfolders contain data from different parts of the experiment and are sorted by participant numbers. For example for participant 1001,I have in each subfolder a file p1001 and similarly for participant 1002, in each folder I have p1002 and so on.. So for each participant, I have three files of the same name but different folders. How can I make a single csv file for each partcipant combining the selected columns from the three different subfolders?

import pandas as pd
import os, csv, pdb
import glob

a=[]
base_dir='/Users/...../Desktop/data/'
folders = ('All', 'Choice', 'Choice_S')
pattern = '{}/[{}]/**/filename.csv'.format(base_dir, ''.join(folders))
for filename in glob.glob(pattern):
    filename=pd.DataFrame
    df1=filename[filename['reaction_time']]
    a.append[df1

This was what i tried doing

Answer 1

I tried to recreate your case with one example

I generated 3 random files each one with 3 columns and 100 lines and each one in a different folder

import numpy as np 
import pandas as pd


a = np.random.rand(100,3) 
b = np.random.rand(100,3) 
c = np.random.rand(100,3) 


dataframe1 = pd.DataFrame(a)
dataframe2 = pd.DataFrame(b)
dataframe3 = pd.DataFrame(c)

dataframe1.columns = dataframe2.columns = dataframe3.columns = {"col1","col2","col3"}

dataframe1.to_csv("./1/a.csv")
dataframe2.to_csv("./2/a.csv")
dataframe3.to_csv("./3/a.csv")

Then i read back the csv files, then for each column of each file I combined the lines and stored the result in a dataframe containing the 300 combined lines of the 3 files using pandas.concat with axis = 0 , then i combined the columns using the same function with axis = 1

a1 = pd.read_csv("./1/a.csv")
a2 = pd.read_csv("./2/a.csv")
a3 = pd.read_csv("./3/a.csv")



combined_col1 = pd.concat([a1["col1"],a2["col1"],a3["col1"]],axis=0) 
combined_col2 = pd.concat([a1["col2"],a2["col2"],a3["col2"]],axis=0) 
combined_col3 = pd.concat([a1["col3"],a2["col3"],a3["col3"]],axis=0) 

combine_col1_col2 = pd.concat([combined_col1,combined_col2],axis=1) 


combine_col1_col2.to_csv("result.csv")

hope it helps.