python combining values column wise for a batch of csv files

Question

I have a batch of more than 100 CSV files of format:

time,0
user time,0.6
CPU load, 5%
swaps,0
etc

I have to merge all of these files - This code gives me the desired output join -1 1 -2 1 -t, -a 1 f1.csv f2.csv >res.csv

time,0,0
user time,0.6,0.8
CPU load, 5%,7%
swaps,0,0
etc

But I can give only 2 files as parameters Is there a way to write a python code to iterate on all csv files in a directory (even a bash file would work)

Answer 1

Something like this? It uses glob to open all.csv files in the current working directory (except for output.csv, if it exists):

def main():

    from pathlib import Path
    from contextlib import ExitStack
    from csv import reader
    from collections import defaultdict

    master_dict = defaultdict(list)

    glob_pattern = "[!output]*.csv"

    with ExitStack() as stack:
        readers = [reader(stack.enter_context(path.open("r"))) for path in Path(".").glob(glob_pattern)]
        for reader in readers:
            for row in reader:
                key, value = row
                master_dict[key].append(value)

    with Path("output.csv").open("w") as master_file:
        for key, value_list in master_dict.items():
            master_file.write(",".join([key] + value_list) + "\n")

    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())