Processing data with same name from multiple directories using python

Question

I am looking for a method to make my python script process data with the same name but located in multiple directories. Specifically I am looking to make images out of the data at different time steps. I have a code that parses the data and creates one image at a time but I would rather have it go through all of the directories and do it all for me rather than doing it manually.

The format of the directories/files is as follows:

Main directory:

• filename: 2Dvizualization.py

Dir1: td.0000000

• filename: sqm-wf-k001-st0001.dx
• filename: sqm-wf-k001-st0001_gs.dx

Dir2: td.0000400

• filename: sqm-wf-k001-st0001.dx
• filename: sqm-wf-k001-st0001_gs.dx

Dir3: td.0000800

• filename: sqm-wf-k001-st0001.dx
• filename: sqm-wf-k001-st0001_gs.dx

. . .

Dir_final: td.016400

• filename: sqm-wf-k001-st0001.dx
• filename: sqm-wf-k001-st0001_gs.dx

and the script I have written to do it manually is:

import numpy as np

import matplotlib.pyplot as plt

//size of the grid for each spatial dimension

sizex = 267

sizey = 267

sizez = 201

//below the data is parsed and made into an array

sqm_data_gs = np.genfromtxt('sqm-wf-k001-st0001_gs.dx',skip_header = 7,skip_footer = 5)

sqm_data = np.genfromtxt('sqm-wf-k001-st0001.dx',skip_header = 7,skip_footer = 5)

sqm_dat_gs = np.array(sqm_data_gs) 

sqm_dat = np.array(sqm_data)

sqm_data_array = np.reshape(sqm_dat,(sizex,sizey,sizez))

sqm_data_array_gs = np.reshape(sqm_dat_gs,(sizex,sizey,sizez))

sqm_diff = sqm_data_array - sqm_data_array_gs


plt.imshow(sqm_diff[:,:,sqm_diff.shape[2]//2],interpolation = 'quadric',origin = 'lower',extent=[-20,20,-20,20])

plt.xlim([-3,3])

plt.ylim([-3,3])

//plt.show()

plt.savefig('sqm0001.png')

I would like to save each of the images to some file called sqm****.png.

Is there a way to loop over all of these directories and output the images as such?

Any insight would be greatly appreciated.

Thank you!

Answer 1

You have to go through your directory tree and find the files you want to process.

In your case I would wrap your actual code in a function and then define another function to find the files and pass them to your processing function.

You can easily walk your directories using os.walk , so this would be an example for your problem:

import os

def find_data_files():
     for root, dirs, files in os.walk('/path/to/your/main/dir'): 
          if len(files) > 0:
              # we are inside a dir with files           
              for file in files:
                  # get file name and extension
                  name, ext = os.path.splitext(file)
                  if ext == '.dx':
                      if name[-3:] == '_gs':
                          # get full path for sql_data_gs file
                          sqm_data_gs = os.path.join(root, file)
                      else:
                          # get full path for sql_data file
                          sqm_data = os.path.join(root, file)
                      # now call your data processing function using file values
                      # (supposing you called your function `process_files)
                      process_files(sqm_data, sqm_data_gs)