将多个 CSV 文件连接到一个 Dataframe 并输出到 Master CSV

Question

I'm looking for someone to help me with the script below. I'm trying to concatenate a month worth of csv files into a 'master file'. The files are really big, so I was hoping to do a few things in the script to shorten them. Here is what I'm having trouble with:

1. The files are different, but the headers are the same. I'm not sure how to get the header on only the first file. I used next(f) to get rid of the rest.
2. How can I add the 'Output' directory as the target folder for output1.csv
3. Lastly, I've been trying to work with pandas- how can I use them to delete columns 1,2,4 and everything after column 90. I also would like to know how to make this a dataframe first before I write it to csv- I would like to add a few calculations to the end of the output file before I write it.

Here's my script so far, Im using the file's timestamp to find the correct month '201510' = October

import csv
import os
import sys
import pandas as pd

Source = r'F:\backup\finalized 2'
Output = r'F:\Tom\Python'

for root, dirs, files in os.walk((os.path.normpath(Source)), topdown=False):
    for name in files:
        if name.startswith('201510') and name.endswith('client.csv'):
            print "Found", name
            SourceFolder = os.path.join(root, name)
            with open(SourceFolder + "", 'r') as f:
                next(f)
                for line in csv.reader(f, delimiter=','):
                    with open('output1.csv','ab') as fout:
                        wr = csv.writer(fout)
                        wr.writerow(line)

Here are the calculations I would like to add to the end of the dataframe/CSV:

df['ten_avg'] = df.iloc[:, 30:50].sum(axis=1).astype('int64') / 20      
df['twenty_avg'] = df.iloc[:, 30:70].sum(axis=1).astype('int64') / 40

Answer 1

I think you can use only pandas for processing.

You need headers of all files, because you need concatenate them to one big files by headers of csvs.

I think better is to define path and name of output file together: OutputCSV = r'F:\Tom\Python\output.csv' .

The best method for reading csv is read only these columns, what exactly need for next processing. You can use function read_csv with parameter usecol . It is filter of columns and need names of columns. You can get them by reading one file with header (all rows can be deleted). Column names in list are processing - deleted 3.item (first item has index 0), slicing them by [2:89] and then use variable cols for reading all csvs.

You get all files in loop, get datafarme from function read_csv with usecols=cols , which is appending to list of dataframes. Then this list is concatenated to one big output dataframe df .

After processing output df is write to file by function to_csv .

import pandas as pd
import os

Source = r'F:\backup\finalized 2'
OutputCSV = r'F:\Tom\Python\output.csv'

normSource = os.path.normpath(Source)

#find column names and delete 1,2,4, and more as 90th columns
#read one csv
names = pd.read_csv(os.path.join(normSource,'header.csv'), sep=",")
#column names to list
cols = names.columns.tolist()
print cols
#the first item has index 0, so you need delete 0, 1, 3, 89, 90, 91.. item

#delete 3 item
del cols[3]

#get 2,4,5,...89 item
cols = cols[2:89]
print cols

dfs = []
#create empty df for output
d = pd.DataFrame()

for root, dirs, files in os.walk(normSource, topdown=False):
    for name in files:
        print root
        print name
        if name.startswith('201510') and name.endswith('client.csv'):
            #only read columns from list cols
            dfs.append(pd.read_csv(os.path.join(root, name), sep=',', index_col=False, usecols=cols))
            df = pd.concat(dfs, ignore_index=True)

#all files in one dataframe
print df.head()

df['ten_avg'] = df.iloc[:, 30:50].sum(axis=1).astype('int64') / 20      
df['twenty_avg'] = df.iloc[:, 30:70].sum(axis=1).astype('int64') / 40
print df.head()

#output to csv, remove index
df.to_csv(OutputCSV, sep=",", index=False)

Answer 2

Pandas Dataframe should come handy here.

Import each file in the dataframe using:

import pandas as pd
df = pd.DataFrame.from_csv('<csvfilename>',index_col=False,parse_dates=False)

And then append this to master dataframe with:

master = master.append(df,ignore_index=True)

Perform your operations on dataframe and then export using:

master.to_csv('<csv_file_name>')