Listing files in shell script

Question

When i try the below code am getting all files whose filename starts with E

#!/bin/bash

data=$(ls -trh E*)
for entry in ${data}
do
  echo ${entry}
done

But if i try the below code , which get the wildcard from argument , i am getting only the first filename

#!/bin/bash

data=$(ls -trh $1)
for entry in ${data}
do
  echo ${entry}
done

Can anyone help me to solve this ..

When i gave quotes like this myscript.sh 'E*' it worked fine , is there any way to do this without giving quotes ?

Answer 1

This is a shell expansion problem.

Your shell will interpret wildcard characters before passing then to your process. For example:

script.sh

#!/bin/bash
echo $1 $2 $3

Running the above with a wildcard:

> ./script.sh E*
Eva   Eve   Evolve

If you want to pass an argument without shell interpreting it first, you will have to quote it:

> ./script.sh 'E*'
E*

Better Solution using find :

What you're actually trying to do is get a list of all files and folders in a given directory, in reverse modification time order (oldest first). The output of ls is notoriously painful to parse. It is preferable to use the powerful and flexible find command instead.

This is a one liner:

>  find ./ -maxdepth 1 -printf "%A@ %f\0" | sort -z -n | while read -d '' date line; do echo "$line"; done

Which is probably cryptic, but makes sense once explained.

• Find all files in this directory without recursing find ./ -maxdepth 1
• For each file, print out their last modified time in second -printf "%A@
• and their filename, separated by null characters %f\\0"
• Sort the null-separated strings by last modified time (numerically) sort -z -n
• For every null separated string assign the timestamp to 'date' and the rest of the line to 'line': while read -d '' date line
• Print the line echo "$line"

For example, in my directory:

> ls -l
total 4296
drwxr-xr-x 2 bfer cvs    4096 2012-03-05 15:49 colortry
drwxr-xr-x 3 bfer cvs    4096 2012-03-27 15:05 githug
drwxr-xr-x 3 bfer cvs    4096 2012-03-12 17:18 hooks-bare
drwxr-xr-x 3 bfer cvs    4096 2012-03-28 12:38 java
-rw-r--r-- 1 bfer cvs 4025413 2012-03-27 12:53 mozdebug
drwxr-xr-x 2 bfer cvs    4096 2012-02-16 12:54 mustache_bug_demo
-rwxr-xr-x 1 bfer cvs     113 2012-03-30 12:20 try.sh
> find ./ -maxdepth 1 -printf "%A@ %f\0" | sort -z -n | while read -d '' date line; do echo "$line"; done
mozdebug
colortry
hooks-bare
mustache_bug_demo
githug
java
try.sh
./

If you don't want the ./ result, just take it out of your final set.

updated: With Sorpigal's suggestion to handle filenames with newlines.

Further note on shell expansion

This is the same with things like parenthesis and ampersands. For example, the following curl commands will not work as expected:

> curl example.com/site?q=hello&name=bob
> echo 23/(7/98) | bc

As the ampersand and parentheses will be interpreted by the shell before they are passed to the process as arguments.

For this to work correctly, you will have to quote the args:

> curl "example.com/site?q=hello&name=bob"
> echo "23/(7/98)" | bc

Answer 2

Did you enclose your wildcard when calling your script?

bash yourscript.sh 'E*'

Otherwise your script will get things that start with E in your current directory and $1 becomes exactly the first file, thus the behavior

Answer 3

This sound like a Shell Expansion problem . If you want to pass wild card to a shell , just quote it . eg .

./script "E*"

Quoting would avoid shell expansion .

Answer 4

You should not be using ls in a shell script. The output of ls is for human consumption only and should never be used as the input to another command.

First, the correct solution to your problem.

#!/usr/bin/env bash
pat="$1"
while IFS= read -r -d '' file ; do
    echo "$file"
done < <(find . -maxdepth 1 -name "$pat" -print0)

This presumes that you don't have any special ordering or name format requirements. You can refer to this answer if you need ordering like ls -t .

Your problem is common and I call it the "Who sees what?" problem. You say myscript.sh E* and expect a literal E* to be passed to your script, but in fact since this argument is unquoted the glob will be expanded by bash before your script is called, so what your script sees is all files in the current directory whose names started with E .

As you note, single-quoting the argument "fixes" this. This is because bash doesn't perform any special expansion inside single quotes, so now your script sees E* literally. You have, effectively, escaped the * so bash won't expand it before passing it to your script. To do this without using quotes is possible by backslash-escape the *

myscript.sh E\*

But a superior solution is to allow bash to expand the glob and change your script to handle processing multiple file arguments instead. For example:

#!/bin/bash

data=("$(ls -trh "$@")")
for entry in "${data[@]}"
do
  echo "${entry}"
done

Here I assume that all arguments will be file names. I have changed data into an array and added quotes to all expansions, to preserve whitespace properly. This solution is still wrong, because it parses ls output, but now you probably get the behavior you expect out of the unquoted E*

myscript.sh E*

In general you should always quote the expansion of variables in shell scripts, because that probably does what you expect (and unquoted expansion probably does not !)

Answer 5

Here's a quick example:

find files of a directory ordered by their modification dates from older to newer but less than or equal a given date and containing a '.inc' in their names

"find-files.sh"
#!/bin/bash
store=$1
date_given=$2
inc_files=()
limit=$( date +"%s" -d "$date_given" )
while read -r -d $'\0' seconds file; do
   seconds=${seconds%.*}
   if [[ $seconds -le $limit ]]; then
      printf "seconds=$seconds, file=$file\n"
      #block word splitting
      inc_files=( "${inc_files[@]}" "$file" )
      #do whatever you want here...
   fi
done < <(find $store -maxdepth 1 -type f -name "*.inc*" -printf "%T@ %f\0" | sort -zn)

and call it:

./find-files.sh '/a/dir/to/search' '2013-01-12 18:12:09'

I agree with @Sorpigal https://unix.stackexchange.com/questions/22674/shell-script-for-moving-oldest-files/29205#29205 it's very helpful except 'IFS=' in while loop that breaks everything in my system (Ubuntu).

Answer 6

Try this.

#!/bin/bash
data=$(ls -trh "$@" | cat -E)
while [ ${#data} != 0 ]
do
  echo ${data%%\$*}
  data=${data#*\$}
done