How to load all .txt files in a folder with separate names - Python

Question

I have to create a bunch of numpy arrays by importing all the.txt files from a folder. This is the way I'm doing it right now:

wil_davide_noIA_IST_nz300           = np.genfromtxt("%s/davide/wil_davide_noIA_IST_nz300.txt" %path)
wig_davide_multiBinBias_IST_nz300   = np.genfromtxt("%s/davide/wig_davide_multiBinBias_IST_nz300.txt" %path)
wig_davide_multiBinBias_PySSC_nz300 = np.genfromtxt("%s/davide/wig_davide_multiBinBias_PySSC_nz300.txt" %path)
wig_davide_noBias_PySSC_nz300       = np.genfromtxt("%s/davide/wig_davide_noBias_PySSC_nz300.txt" %path)
wig_davide_IST_nz300                = np.genfromtxt("%s/davide/wig_davide_IST_nz300.txt" %path)
wig_davide_noBias_IST_nz300         = np.genfromtxt("%s/davide/wig_davide_noBias_IST_nz300.txt" %path)
wig_davide_PySSC_nz300              = np.genfromtxt("%s/davide/wig_davide_PySSC_nz300.txt" %path)
...

from the folder

Can I automate the process somehow? Note that I'd like the array to have the same name as the imported file (without the.txt, of course).
Thank you very much

Answer 1

This is how you can get all the filenames.

path is whatever is your file path ( "./" if in the same directory)

import os

path = "./"

filenames = []
for filename in os.listdir(path):
  filenames.append[file]

...

Also, you can filter some files with the if-structure provided. Here I'm selecting only txt files.

filter is some filter you can apply (use "." not in file to get directories)

import os

path = "./"
filter = ".txt"

txt_filenames = []
for filename in os.listdir(path):
  if filter in file:
    txt_filenames.append[filename]

...

Answer 2

A heads up: You should not do this, if there are any security concerns with the files or their names.

Otherwise you can use the exec function to get the variables you want. For a list approach see the other solution.

import os
path_to_folder = "."
command = ""
for f in os.listdir(path_to_folder):
    if f.endswith(".txt"):
        command += f.replace(".txt", "") + f" = np.genfromtxt('{path_to_folder}/{f}')\n"
exec(command)

Depended on your folder/file structure you would need to change up the code a bit, but that should work like asked.

Answer 3

I haven't tested this, but I think this pseudo-code should work - the only thing "pseudo" about it is the hardcoded "dir/to/files" string, which you'll have to change to the path to the directory containing the text files. In modern Python you would use the pathlib or glob standard library modules to iterate over all text files in a given directory. Creating a variable number of variables, with variable names determined at runtime is never a good idea. Instead, keep your numpy arrays in some kind of collection. In this case, I would suggest a dictionary, so that you can access the individual numpy arrays by key, where the key is the name of the corresponding file as a string - without the extension of course:

def get_kv_pairs():
    from pathlib import Path
    for path in Path("dir/to/files").glob("*.txt"):
        yield path.stem, np.genfromtxt(str(path))
        

arrays = dict(get_kv_pairs())

print(arrays["wil_davide_noIA_IST_nz300"]) # Here's how you would access the individual numpy arrays.