遍历子目录中的文件并加载它们

Question

I've got the following situation to solve: my parent directory has ~25 subfolders. Each of the subfolders contains a file that ends with 'RRS.csv'. Additonally, some of the subfolders contain a file ending with 'ROH.csv'. From each of the subfolders, i need to import the 'ROH.csv' file if it exists, and if not, the 'RRS.csv' file. I tried this through iterating through all subfolders and all files in the subfolders using the os.path.exists operator to check if the 'ROH.csv' file exists. Another idea was to first list all files in each subfolder, then identifying if one element contains the 'ROH.csv' ending and then loading it.

for filename in sorted(os.listdir(parent_dir)):
    for subdir, dirs, files in os.walk(parent_dir):
        if not os.path.exists(filename.endswith('ROH.csv')):
            data = np.genfromtxt(filename.endswith('RRS.csv'), delimiter=',', skip_header=1)
            # some calculations
        else:
            data = np.genfromtxt(filename.endswith('ROH.csv'), delimiter=',', skip_header=1)
            # more funny calculations

This code has multiple problems: (i) it has to check if one file in the subfolder ends with 'ROH.csv', and not if each file ends with it; (ii) i havent figured out a way yet to specify which file to load; endswith does not work (bool); (iii) it contains double for-loops.

Hope anyone has an idea to solve this.

Answer 1

I'm not really sure I understand the problems you mention, but here is a snippet that gives you the list of file to process. You can change the filtering in more clever or readable ways, but I have tested this with one level of nested folders and it does the job. You can write for sure something more clear or suitable to your needs starting from here

#parent_dir=os.getcwd()+"\\temp" 
files_to_read=[]
walk = [(subdir, dirs, files) for (subdir, dirs, files) in os.walk(parent_dir) if not (subdir==parent_dir)]  #Skip the root directory
for (subdir, dirs, files) in walk:
    file_to_read = list(filter(lambda x: "ROH.csv" in x or "RRS.csv" in x, files)) #Explicitly filter for one of the two strings
    if len(file_to_read)>1:
        file_to_read = list(filter(lambda x: "ROH.csv" in x, files))[0]      #explictly pick the ROH file if there are both files
    elif len(file_to_read)>0:
        file_to_read = file_to_read[0]                           #Otherwise pick the only file in the list i.e. RRS
    file_to_read=subdir+os.path.sep+file_to_read
    files_to_read.append(file_to_read)

Answer 2

This is assuming you need to do different things for ROH files and RRS files:

def get_files_with_suffix(root, files, suffix):
    return [os.path.join(root, filename) for filename in files if filename.endswith(suffix)]

for root, dirs, files in os.walk(parent_dir):
    roh_files = get_files_with_suffix(root, files, 'ROH.csv')
    if roh_files:
        for roh_file in roh_files:
            data = np.genfromtxt(roh_file, delimiter=',', skip_header=1)
            # more funny calculations
    else:
        rrs_files = get_files_with_suffix(root, files, 'RRS.csv')
        for rrs_file in rrs_files:
            data = np.genfromtxt(rrs_file, delimiter=',', skip_header=1)
            # some calculations

If the calculations are the same, I would extract all the code duplication to different functions:

def give_this_a_better_name_that_explains_the_specific_calculations(filename):
    data = np.genfromtxt(filename, delimiter=',', skip_header=1)
    # some calculations

def get_files_with_suffix(root, files, suffix):
    return [os.path.join(root, filename) for filename in files if filename.endswith(suffix)]

def process_files_with_suffix_but_give_this_a_better_name_too(root, files, suffix):
    files = get_files_with_suffix(root, files, suffix)
    for file in files:
        give_this_a_better_name_that_explains_the_specific_calculations(file)
    return files

for root, dirs, files in os.walk(parent_dir):
    if not process_files_with_suffix_but_give_this_a_better_name_too(root, filename, 'ROH.csv'):
        process_files_with_suffix_but_give_this_a_better_name_too(root, filename, 'RRS.csv')