在 Mac 上使用 Python 在子文件夹中循环检查文件是否存在，打开、复制、粘贴然后关闭文件

Question

I have a folder, with numerous (~200) sub-folders on my Mac desktop.

Some (but not all) sub-folders contain a csv file named "sample.csv".

Further, I have a "aggregate.csv" file where I would like to copy the 2nd column of each "sample.csv" data into.

Structure:

"/Desktop/folder"
"/Desktop/folder/aggregate.csv"

"/Desktop/folder/sub-folder"
"/Desktop/folder/sub-folder/sample.csv"

Using Python, how can I loop though each sub-folder, check if "sample.csv" exists, open it, copy the 2nd column, paste this column into the "aggregate.csv" file, close "sample.csv", then move on to the next sub-folder?

In "aggregate.csv", the copied data should increment to the right, so it doesn't overwrite the previous "sample.csv" data that has just been pasted.

My computer is opening the CSV files with Excel, that's why I refer to the "2nd column".

Many thanks

Answer 1

$ cd ~

$ more aggregate.csv
X
X
X
X
X
X

$ more ./Desktop/folder/sub-folder/sample.csv
A,1
A,2
A,3
A,4
A,5

$ more ./Desktop/folder/sub-folder/sub-sub-folder/sample.csv
B,6
B,7
B,8
B,9

$ more ./Desktop/folder/sub-folder2/sample.csv
C,10
C,11
C,12
C,13
C,14
C,15
C,16

$ more ./Desktop/folder/sub-folder3/sub-sub-folder/sample.csv
D,17
D,18
D,19

$ python3 aggregate_samples.py ./Desktop
./Desktop/folder/sub-folder/sample.csv
./Desktop/folder/sub-folder/sub-sub-folder/sample.csv
./Desktop/folder/sub-folder2/sample.csv
./Desktop/folder/sub-folder3/sub-sub-folder/sample.csv

$ cat aggregate.csv

X,1,6,10,17
X,2,7,11,18
X,3,8,12,19
X,4,9,13,
X,5,,14,
X,,,15,
,,,16,

Here is the code that accomplishes this. The key technologies you need: os.walk() to recursively search the folders, the csv module to read in the sample.csv files (and get the 2nd column), lists to accumulate the samples, and csv again to write out the result. I assumed your sample.csv files will be different lengths, and so the code handles that (by pre-allocating a sparse matrix).

This assumes your dataset is small enough to fit into memory. If not, then more work needs to be done.

# aggregate_samples.py
import os
import sys
import argparse
import csv

def main(options):
    columns = []

    try:
        # Load in aggregate.csv, if there is one.
        with open('aggregate.csv') as f:
            column = [line.rstrip('\n') for line in f]
            columns.append(column)
    except FileNotFoundError:
        # Doesn't exist; create it later.
        pass

    longest_sample = 0
    for d, subdirs, files in os.walk(options.directory):
        subdirs.sort()
        for filename in files:

            if filename == 'sample.csv':
                file_path = os.path.join(d, filename)
                print(file_path)

                samples = []
                with open(file_path) as f:
                    reader = csv.reader(f, delimiter=',')
                    # Get the 2nd column.
                    for sample in reader:
                        samples.append(sample[1])
                longest_sample = max(longest_sample, len(samples))
                columns.append(samples)

    # Pre-fill a transpose matrix according to number of columns
    # and longest colum.
    a = [ [ '' for i in columns ] for j in range(longest_sample) ]

    # Move samples into matrix, transposing as you go.
    for i in range(len(columns)):
        for j in range(len(columns[i])):
            a[j][i] = columns[i][j]

    # Output matrix as CSV.
    with open('aggregate.csv', 'w+') as aggregate:
        writer = csv.writer(aggregate, delimiter=',')
        writer.writerows(a)

    return 0

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument(
        'directory',
        help='Directory path.')
    options = parser.parse_args()
    sys.exit(main(options))

Answer 2

You can get all the info you need about the script package here .

Here is how to set the Program argument, using Packages->Script->Configure Script: