从起始行合并多个csv

Question

I wanted to know if I can combine multiple CSV but to starting from a given row and add the name of the file on the first column. Currently, I have been using the following code:

import os

import glob

import pandas as pd

os.chdir(Path)

extension = 'csv'

all_filenames = [i for i in glob.glob('*.{}'.format(extension))]

#combine all files in the list
combined_csv = pd.concat([pd.read_csv(f) for f in all_filenames ])
#export to csv
combined_csv.to_csv( "combined.csv", index=False, encoding='utf-8-sig')

Now I am dealing with some files that are not well-formatted, and I need to start concatenating from row 3 and add the name of the file on the first column but not sure how to make that happen

Answer 1

(First of all you don't need the list comprehension with glob.glob - it returns a list anyway.)

For your request to concatenate all files only from row 3 on - this should be fairly simple. Just add a .loc[3:] to your read_csv command:

combined_csv = pd.concat([pd.read_csv(f).loc[3:] for f in all_filenames ])

But if you want to make more modifications, you should use a normal for loop, to get all things done in a readable way and concat everything afterwards, like:

mydfs = []
for f in all_filenames:
    df = pd.read_csv(f).loc[3:]
    df['filename'] = f
    mydfs.append(df)

combined_csv = pd.concat(mydfs)