Cannot print entire array in Bash Shell script

Question

I've written a shell script to get the PIDs of specific process names (eg pgrep python , pgrep java ) and then use top to get the current CPU and Memory usage of those PIDs.

I am using top with the '-p' option to give it a list of comma-separated PID values. When using it in this mode, you can only query 20 PIDs at once, so I've had to come up with a way of handling scenarios where I have more than 20 PIDs to query. I'm splitting up the list of PIDs passed to the function below and "despatching" multiple top commands to query the resources:

# $1 = List of PIDs to query
jobID=0
for pid in $1; do
    if [ -z $pidsToQuery ]; then
        pidsToQuery="$pid"
    else
        pidsToQuery="$pidsToQuery,$pid"
    fi
    pidsProcessed=$(($pidsProcessed+1))
    if [ $(($pidsProcessed%20)) -eq 0 ]; then
        debugLog "DESPATCHED QUERY ($jobID): top -bn 1 -p $pidsToQuery | grep \"^ \" | awk '{print \$9,\$10}' | grep -o '.*[0-9].*' | sed ':a;N;\$!ba;s/\n/ /g'"
        resourceUsage[$jobID]=`top -bn 1 -p "$pidsToQuery" | grep "^ " | awk '{print $9,$10}' | grep -o '.*[0-9].*' | sed ':a;N;$!ba;s/\n/ /g'`
        jobID=$(($jobID+1))
        pidsToQuery=""
    fi
done
resourceUsage[$jobID]=`top -bn 1 -p "$pidsToQuery" | grep "^ " | awk '{print $9,$10}' | grep -o '.*[0-9].*' | sed ':a;N;$!ba;s/\n/ /g'`

The top command will return the CPU and Memory usage for each PID in the format (CPU, MEM, CPU, MEM etc)...:

13 31.5 23 22.4 55 10.1

The problem is with the resourceUsage array. Say, I have 25 PIDs I want to process, the code above will place the results of the first 20 PIDs in to $resourceUsage[0] and the last 5 in to $resourceUsage[1] . I have tested this out and I can see that each array element has the list of values returned from top.

The next bit is where I'm having difficulty. Any time I've ever wanted to print out or use an entire array's set of values, I use ${resourceUsage[@]} . Whenever I use that command in the context of this script, I only get element 0's data. I've separated out this functionality in to a script below, to try and debug. I'm seeing the same issue here too (data output to debug.log in same dir as script):

#!/bin/bash

pidList="1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25"

function quickTest() {
    for ((i=0; i<=1; i++)); do
        resourceUsage[$i]=`echo "$i"`
    done

    echo "${resourceUsage[0]}"
    echo "${resourceUsage[1]}"
    echo "${resourceUsage[@]}"
}


function debugLog() {
    debugLogging=1
    if [ $debugLogging -eq 1 ]; then
        currentTime=$(getCurrentTime 1)
        echo "$currentTime - $1" >> debug.log
    fi
}

function getCurrentTime() {
    if [ $1 -eq 0 ]; then
        echo `date +%s`
    elif [ $1 -eq 1 ]; then
        echo `date`
    fi
}

jobID=0

for pid in $pidList; do
    if [ -z $pidsToQuery ]; then
        pidsToQuery="$pid"
    else
        pidsToQuery="$pidsToQuery,$pid"
    fi
    pidsProcessed=$(($pidsProcessed+1))
    if [ $(($pidsProcessed%20)) -eq 0 ]; then
        debugLog "DESPATCHED QUERY ($jobID): top -bn 1 -p $pidsToQuery | grep \"^ \" | awk '{print \$9,\$10}' | grep -o '.*[0-9].*' | sed ':a;N;\$!ba;s/\n/ /g'"
        resourceUsage[$jobID]=`echo "10 10.5 11 11.5 12 12.5 13 13.5"`
        debugLog "Resource Usage [$jobID]: ${resourceUsage[$jobID]}"
        jobID=$(($jobID+1))
        pidsToQuery=""
    fi
done
#echo "Dispatched job: $pidsToQuery"
debugLog "DESPATCHED QUERY ($jobID): top -bn 1 -p $pidsToQuery | grep \"^ \" | awk '{print \$9,\$10}' | grep -o '.*[0-9].*' | sed ':a;N;\$!ba;s/\n/ /g'"
resourceUsage[$jobID]=`echo "14 14.5 15 15.5"`
debugLog "Resource Usage [$jobID]: ${resourceUsage[$jobID]}"
memUsageInt=0
memUsageDec=0
cpuUsage=0
i=1
debugLog "Row 0: ${resourceUsage[0]}"
debugLog "Row 1: ${resourceUsage[1]}"
debugLog "All resource usage results: ${resourceUsage[@]}"
for val in ${resourceUsage[@]}; do
    resourceType=$(($i%2))
    if [ $resourceType -eq 0 ]; then
        debugLog "MEM RAW: $val"
        memUsageInt=$(($memUsageInt+$(echo $val | cut -d '.' -f 1)))
        memUsageDec=$(($memUsageDec+$(echo $val | cut -d '.' -f 2)))
        debugLog "   MEM INT: $memUsageInt"
        debugLog "   MEM DEC: $memUsageDec"
    elif [ $resourceType -ne 0 ]; then
        debugLog "CPU RAW: $val"
        cpuUsage=$(($cpuUsage+$val))
        debugLog "CPU TOT: $cpuUsage"
    fi
    i=$(($i+1))
done
debugLog "$MEM DEC FINAL: $memUsageDec (pre)"
memUsageDec=$(($memUsageDec/10))
debugLog "$MEM DEC FINAL: $memUsageDec (post)"
memUsage=$(($memUsageDec+$memUsageInt))
debugLog "MEM USAGE: $memUsage"
debugLog "CPU USAGE: $cpuUsage"
debugLog "MEM USAGE: $memUsage"
debugLog "PROCESSED VALS: $cpuUsage,$memUsage"
echo "$cpuUsage,$memUsage"

I'm really stuck here as I've printed out entire arrays before in Bash Shell with no problem. I've even repeated this in the shell console with a few lines and it works fine there:

listOfValues[0]="1 2 3 4"
listOfValues[1]="5 6 7 8"
echo "${listOfValues[@]}"

Am I missing something totally obvious? Any help would be greatly appreciated!

Thanks in advance! :)

Answer 1

Welcome to StackOverflow, and thanks for providing a test case! The bash tag wiki has additional suggestions for creating small, simplified test cases. Here's a minimal version that shows your problem:

log() {
  echo "$1"
}
array=(foo bar)
log "Values: ${array[@]}"

Expected: Values: foo bar . Actual: Values: foo .

This happens because ${array[@]} is magic in quotes, and turns into multiple arguments. The same is true for $@ , and for brevity, let's consider that:

Let's say $1 is foo and $2 is bar .

• The single parameter "$@" (in quotes) is equivalent to the two arguments "foo" "bar" .
• "Values: $@" is equivalent to the two parameters "Values: foo" "bar"

Since your log statement ignores all arguments after the first one, none of them show up. echo does not ignore them, and instead prints all arguments space separated, which is why it appeared to work interactively.

This is as opposed to ${array[*]} and $* , which are exactly like $@ except not magic in quotes, and does not turn into multiple arguments.

• "$*" is equivalent to "foo bar"
• "Values: $*" is equivalent to "Values: foo bar"

In other words: If you want to join the elements in an array into a single string, Use * . If you want to add all the elements in an array as separate strings, use @ .

Here is a fixed version of the test case:

log() {
  echo "$1"        
}
array=(foo bar)
log "Values: ${array[*]}"

Which outputs Values: foo bar

Answer 2

I would use ps , not top , to get the desired information. Regardless, you probably want to put the data for each process in a separate element of the array, not one batch of 20 per element. You can do this using a while loop and a process substitution. I use a few array techniques to simplify the process ID handling.

pid_array=(1 2 3 4 5 6 7 8 9 ... )
while (( ${#pid_array[@]} > 0 )); do
     printf -v pidsToQuery "%s," "${pid_array[@]:0:20}"
     pid_array=( "${pid_array[@]:20}" )

     while read cpu mem; do
         resourceUsage+=( "$cpu $mem" )
     done < <( top -bn -1 -p "${pidsToQuery%,}" ... )
done