Bash File names will not append to file from script

Question

Hello I am trying to get all files with Jane's name to a separate file called oldFiles.txt. In a directory called "data" I am reading from a list of file names from a file called list.txt, from which I put all the file names containing the name Jane into the files variable. Then I'm trying to test the files variable with the files in list.txt to ensure they are in the file system, then append the all the files containing jane to the oldFiles.txt file(which will be in the scripts directory), after it tests to make sure the item within the files variable passes.

#!/bin/bash

> oldFiles.txt
files= grep " jane " ../data/list.txt | cut -d' ' -f 3
if test -e ~data/$files; then
  for file in $files; do
    if test -e ~/scripts/$file; then
      echo $file>> oldFiles.txt
    else
      echo "no files"
    fi
  done
fi

The above code gets the desired files and displays them correctly, as well as creates the oldFiles.txt file, but when I open the file after running the script I find that nothing was appended to the file. I tried changing the file assignment to a pointer instead files= grep " jane "../data/list.txt | cut -d' ' -f 3 ---> files=$(grep " jane "../data/list.txt) files= grep " jane "../data/list.txt | cut -d' ' -f 3 ---> files=$(grep " jane "../data/list.txt) to see if that would help by just capturing raw data to write to file, but then the error comes up "too many arguments on line 5" which is the 1st if test statement. The only way I get the script to work semi-properly is when I do ./findJane.sh > oldFiles.txt on the shell command line, which is me essentially manually creating the file. How would I go about this so that I create oldFiles.txt and append to the oldFiles.txt all within the script?

Answer 1

As far as I can tell, this is what you're going for. This is totally a community effort based on the comments, catching your bugs. Obviously credit to Mark and Jetchisel for finding most of the issues. Notable changes:

• Fixed $files to use command substitution
• Fixed path to data/$file , assuming you have a directory at ~/data full of files
• Fixed the test to not test for a string of files, but just the single file (also using -f to make sure it's a regular file)
• Using double brackets — you could also use double quotes instead, but you explicitly have a Bash shebang so there's no harm in using Bash syntax
• Adding a second message about not matching files, because there are two possible cases there; you may need to adapt depending on the output you're looking for
• Removed the initial empty redirection — if you need to ensure that the file is clear before the rest of the script, then it should be added back, but if not, it's not doing any useful work
• Changed the shebang to make sure you're using the user's preferred Bash, and added set -e because you should always add set -e

#!/usr/bin/env bash
set -e

files=$(grep " jane " ../data/list.txt | cut -d' ' -f 3)
for file in $files; do
    if [[ -f $HOME/data/$file ]]; then
        if [[ -f $HOME/scripts/$file ]]; then
            echo "$file" >> oldFiles.txt
        else
            echo "no matching file"
        fi
    else
        echo "no files"
    fi
done

Answer 2

The biggest problem you have is matching names like "jane" or "Jane's" , etc. while not matching "Janes" . grep provides the options -i (case insensitive match) and -w (whole-word match) which can tailor your search to what you appear to want without having to use the kludge ( " jane " ) of appending spaces before an after your search term. (to properly do that you would use [[:space:]]jane[[:space:]] )

You also have the problem of what is your "script dir" if you call your script from a directory other than the one containing your script, such as calling your script from your $HOME directory with bash script/findJane.sh . In that case your script will attempt to append to $HOME/oldFiles.txt . The positional parameter $0 always contains the full pathname to the current script being run, so you can capture the script directory no matter where you call the script from with:

dirname "$0"

You are using bash, so store all the filenames resulting from your grep command in an array, not some general variable (especially since your use of " jane " suggests that your filenames contain whitespace)

You can make your script much more flexible if you take the information of your input file (eg list.txt ), the term to search for (eg "jane" ), the location where to check for existence of the files (eg $HOME/data ) and the output filename to append the names to (eg "oldFile.txt" ) as command line [positonal] parameters. You can give each default values so it behaves as you currently desire without providing any arguments.

Even with the additional scripting flexibility of taking the command line arguments, the script actually has fewer lines simply filling an array using mapfile (synonymous with readarray ) and then looping over the contents of the array. You also avoid the additional subshell for dirname with a simple parameter expansion and test whether the path component is empty -- to replace with '.' , up to you.

If I've understood your goal correctly, you can put all the pieces together with:

#!/bin/bash

# positional parameters
src="${1:-../data/list.txt}"  # 1st param - input (default: ../data/list.txt)
term="${2:-jane}"             # 2nd param - search term (default: jane)
data="${3:-$HOME/data}"       # 3rd param - file location (defaut: ../data)
outfn="${4:-oldFiles.txt}"    # 4th param - output (default: oldFiles.txt)

# save the path to the current script in script
script="$(dirname "$0")"

# if outfn not given, prepend path to script to outfn to output
# in script directory (if script called from elsewhere)
[ -z "$4" ] && outfn="$script/$outfn"

# split names w/term into array
# using the -iw option for case-insensitive whole-word match
mapfile -t files < <(grep -iw "$term" "$src" | cut -d' ' -f 3)

# loop over files array
for ((i=0; i<${#files[@]}; i++)); do
  # test existence of file in data directory, redirect name to outfn
  [ -e "$data/${files[i]}" ] && printf "%s\n" "${files[i]}" >> "$outfn"
done

(note: test expression and [ expression ] are synonymous, use what you like, though you may find [ expression ] a bit more readable)

(further note: "Janes" being plural is not considered the same as the singular -- adjust the grep expression as desired)

Example Use/Output

As was pointed out in the comment, without a sample of your input file, we cannot provide an exact test to confirm your desired behavior.

Let me know if you have questions.