根据文件夹名称创建数组

Question

I have data that has been collected and organized in multiple folders.
In each folder, there can be multiple similar runs -- eg collected data under the same conditions, at different times. These filenames contain a number in them that increments. Each folder contains similar data collected under different conditions. For example, I can have an idle folder , and in it can be files named idle_1.csv , idle_2.csv , idle_3.csv , etc. Then I can have another folder pos1 folder , and similarly, pos1_1.csv , pos1_2.csv , etc.

In order to keep track of what folder and what file the data in the arrays came from, I want to use the folder name, "idle", "pos1", etc, as the array name. Then, each file within that folder (or the data resulting from processing each file in that folder, rather) becomes another row in that array.

For example, if the name of the folder is stored in variable arrname, and the file index is stored in variable arrndx, I want to write the value into that array:

arrname[arrndx]=value

This doesn't work, giving the following error:

TypeError: 'str' object does not support item assignment

Then, I thought about using a dictionary to do this, but I think I still would run into the same issue. If I use a dictionary, I think I need each dictionary's name to be the name derived from the folder name -- creating the same issue. If I instead try to use it as a key in a dictionary, the entries get overwritten with data from every file from the same folder since the name is the same:

    arrays['name']=arrname
    arrays['index']=int(arrndx)
    arrays['val']=value

    arrays['name': arrname, 'index':arrndx, 'val':value]

I can't use 'index' either since it is not unique across each different folder.

So, I'm stumped. I guess I could predefine all the arrays, and then write to the correct one based on the variable name, but that could result in a large case statement (is there such a thing in python?) or a big if statement. Maybe there is no avoiding this in my case, but I'm thinking there has to be a more elegant way...

EDIT

I was able to work around my issue using globals():

globals()[arrname].insert(int(arrndx),value)

However, I believe this is not the "correct" solution, although I don't understand why it is frowned upon to do this.

Answer 1

Use a nested dictionary with the folder names at the first level and the file indices (or names) at the second.

from pathlib import Path

data = {}
base_dir = 'base'
for folder in Path(base_dir).resolve().glob('*'):
    if not folder.is_dir():
        continue
    data[folder.name] = {}
    for csv in folder.glob('*.csv'):
        file_id = csv.stem.split('_')[1]
        data[folder.name][file_id] = csv

The above example just saves the file name in the structure but you could alternatively load the file's data (eg using Pandas) and save that to the dictionary. It all depends what you want to do with it afterwards.

Answer 2

What about :

foldername = 'idle'  # Say your folder name is idle for example 
files = {}
files[filename] = [filenmae + "_" + str(i) + ".csv" for i in range(1, number_of_files_inside_folder + 2)]

does that solve your problem ?