How to check the status of bash shell script while executing from Python script?

Question

I have a simple Python script which will execute a shell script using subprocess module in Python.

Below is my Python shell script which is calling testing.sh shell script and it works fine.

import os
import json
import subprocess

jsonData = '{"pp": [0,3,5,7,9]}'
jj = json.loads(jsonData)

os.putenv( 'jj3', ' '.join( str(v) for v in jj['pp']  ) )

print "start"
proc = subprocess.Popen('testing.sh', stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout, stderr) = proc.communicate()
if stderr:
   print "Shell script gave some error"
   print stderr
else:
   print stdout
   print "end" # Shell script ran fine.

And below is my testing.sh shell script -

#!/bin/bash

dir1=some_directory
dir2=some_directory

length1=some_number
length2=some_number

if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
    for el in $jj3
    do
        scp david@machineB:/data/be_t1_snapshot/20131215/t1_"$el"_5.data /data01/primary/. || scp david@machineC:/data/be_t1_snapshot/20131215/t1_"$el"_5.data /data01/primary/.
    done        
fi

What my above shell script does is, it will copy the files from machineB OR machineC to machineA . If the files are not there in machineB then it should be there in machineC always. So it will try to copy the files from machineB to machineA but if the files are not there in machineB then it will try to copy the files from machineC to machineA .

Now my above Python script (which I am running from machineA ) tries to execute my above shell script and see whether my script got executed successfully or not. As I am storing the stderr of my shell script and stdout as well.

Problem Statement:-

With the above approach there is one problem that I am seeing. As I mentioned if the files are not there in machineB , then it will try to copy the files from machineC to machineA . So whenever I run my Python script which calls my shell script, what happens is that, some files are not there in machineB so it throws some exception and then it tries copying the files from machineC but when the call comes back to Python script, it always go inside if stderr: block as there was an error while copying the files from machineB (which is not what I want) and end print statement doesn't gets printed out.

So the question is if the files are not there in machineB , it will try copying the files from machineC and if the file is there is machineC and got successfully copied to machineA without any error, then I want to call that as a success instead of failure . In current scenario what is happening, if the files are not there in machineB and got successfully copied from machineC then it still counts as a failure and end print statement doesn't gets printed out.

I also want to see whether my shell script has any problem while executing from Python. If yes, then I don't want to print end statement.

How should I overcome this problem?

UPDATED:-

What will happen with the below shell script? As the first for loop will failed because david is not a linux command, but my scp command works fine. So still I will see status code as not equal to 0 ?

for i in $( david ); do
    echo item: $i
done

dir1=some_directory
dir2=some_directory

length1=some_number
length2=some_number

if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
    for el in $jj3
    do
        scp david@machineB:/data/be_t1_snapshot/20131215/t1_"$el"_5.data /data01/primary/. || scp david@machineC:/data/be_t1_snapshot/20131215/t1_"$el"_5.data /data01/primary/.
    done        
fi

Answer 1

A process which worked correctly can still write to stderr , so it is wrong to check for problems using if stderr:

Instead you should check the processes exit code. If it is not equal to zero, there was an error, eg:

[...]
if proc.returncode != 0:
    print "Shell script gave some error"
[...]