iteratively move through subdirectories and execute commands on all files within

Question

I am trying to make a script that will from a top directory enter each subdirectory, make a few directories, run commands on each file in the subdirectory, then rename and move around the files. I need it to do this without any additional input from me, like giving the name of the directory or anything, because it pretty much has to run on thousands of directories. This is the script I wrote, which I really don't actually understand:

#!/bin/bash
for path in ./*;
do
[ -d "${path}" ] || continue
dirname="$(basename "${path}")"
cd $path
mkdir stats
mkdir dir
files=(./*)
for f in "${files[@]}"
do
do stuff
done
cd dir
mkdir stats
mv *.trimmed_segments* stats/
rename '.fastq.trimmed' '.fastq' ./*
cd ..
mkdir raw
mv *.fastq raw/
mv trimmed/*.fastq ./
cd ..
done

It will seem to work on a few subdirectories, strangely enough, and then fritz out and start sending a bunch of errors about the perl scripts that are fairly nonspecific.

The commands themselves are fine, if you go into the folder and enter them manually there are no problems, so there is some kind of problem with throwing the right variables around. If you can see my error I would appreciate it so much.

Answer 1

Have you considered using find with the exec option?

I typically use it when I need to "walk through" a path.

Play around with the min/max depth values as you see fit. You can also execute a script that will run a set of commands.

#Directories
find ./ -mindepth 1 -maxdepth 1 -type d -exec bash -c 'mkdir {}/stats; mkdir {}/dir' \;

#Files
find ./ -maxdepth 1 -type f -exec bash -c 'commands here...' \;

Answer 2

I don't think this is an answer per se, but it will help you and future answerers debug, and I'll update it if there's enough other information.

Here's your script with indentation and comments:

#!/bin/bash
for path in ./*; do
  [ -d "${path}" ] || continue
  dirname="$(basename "${path}")"
  cd $path                               # origin/$path
  mkdir stats                            # (this seems not to be used)
  mkdir dir
  files=(./*)
  for f in "${files[@]}"; do
    # do stuff
  done
  cd dir                                 # origin/$path/dir
  mkdir stats
  mv *.trimmed_segments* stats/
  rename '.fastq.trimmed' '.fastq' ./*
  cd ..                                  # origin/$path
  mkdir raw
  mv *.fastq raw/
  mv trimmed/*.fastq ./
  cd ..                                  # origin
done

One technique you might want to consider is to use ( and ) as grouping commands to create subshells , which will also encapsulate any changes to the working directory. As it stands, you use a lot of relative paths with cd , and that can make it hard to tell exactly where you are—especially if any command fails. (To that end, use set -e to exit immediately when any command exits with an error code, which may help you identify where any errors happen.)

#!/bin/bash
set -e                                   # fail immediately on error
for path in ./*; do
  [ -d "${path}" ] || continue
  dirname="$(basename "${path}")"
  (
    cd $path                             # origin/$dirname
    mkdir stats
    mkdir dir
    files=(./*)
    for f in "${files[@]}"; do
      # do stuff
    done
    (
      cd dir                             # origin/$dirname/dir
      mkdir stats
      mv *.trimmed_segments* stats/
      rename '.fastq.trimmed' '.fastq' ./*
    )
    mkdir raw
    mv *.fastq raw/
    mv trimmed/*.fastq ./
  )
done

Answer 3

Another technique if it's a tree of arbitrary depth is to make a function, and call it recursively (that is, from within itself; when it completes each time, it will return to where it was called).

I haven't actually tried this, but it would look something like:

processDir () {
    cd "$1"
    # it's a directory, so what you need to for a directory here
    # ...
    for path in ./*; do
        if [[ -d "${path}" ]]; then
            processDir "${path}"
        else
            # it's a file, not a dir, so do file operations here, if any
            # ...
        fi
    done
}

processDir yourStartDirectory