Using python, how to read all the CSV files in a folder and write the content of the same to a new single CSV file?

Question

I have a set of CSV files in a folder named " data ," these file names are something like this: AA001.csv ; AA002.csv ; BB001.csv ; BB002.csv ; and so on...

The content in the file AA001.csv is as shown below. The content in other CSV files in this folder will be similar to this.

date        level     class
2020-01-01  321       low
2020-01-02  984.2     medium

I have a CSV file named " main.csv " in another folder, whose content is something of this kind:

code    continent    
AA001   Europe
AA002   Asia
.
.

Here, The first column in the main.csv contains the names of the CSV files in the folder " data " that we have seen before. Now, I want to read the data from the CSV files in the folder " data " and write the content of each CSV file in main.csv corresponding to its file name. The example of the expected output:

code    continent    date        level     class
AA001   Europe       2020-01-01  321       low
AA001   Europe       2020-01-02  984.2     medium
AA002   Asia 
.
.

Any help in this regard is highly appreciated.

Answer 1

You could use pd.concat to join the csv files and then merge with your main csv. Sample code here :

import pandas as pd
import os

df = pd.read_csv('main.csv')

print('What df looks like:')
print(df)

dir_name = 'data'

csvs = pd.concat([pd.read_csv(os.path.join(dir_name, fl)).assign(code=os.path.splitext(fl)[0]) for fl in os.listdir(dir_name)])

print('\nWhat csvs look like:')
print(csvs)

df = df.merge(csvs, how='outer')

print('\nWhat merged df looks like:')
print(df)

# export result to csv
df.to_csv('result.csv')

Result:

What df looks like:
    code continent
0  AA001    Europe
1  AA002      Asia

What csvs look like:
         date  level   class   code
0  2020-01-01  321.0     low  AA001
1  2020-01-02  984.2  medium  AA001
0  2020-02-01  456.0    high  AA002
1  2020-02-02  789.0     NaN  AA002

What merged df looks like:
    code continent        date  level   class
0  AA001    Europe  2020-01-01  321.0     low
1  AA001    Europe  2020-01-02  984.2  medium
2  AA002      Asia  2020-02-01  456.0    high
3  AA002      Asia  2020-02-02  789.0     NaN