Executing a shell script in Python from the JSON document

Question

I am trying to execute the shell script in Python using subprocess module.

Below is my shell script which is called as testing.sh .

#!/bin/bash

hello=$jj1

echo $hello

echo $jj1
echo $jj2

for el1 in $jj3
do
    echo "$el1"
done

for el2 in $jj4
do
    echo "$el2"
done

Now I am trying to execute the above shell script in Python so I did like this -

subprocess.call(['./testing.sh'])

and it works fine. Now I am thinking to add the above script in a JSON document like this and then execute it -

json_script = '{"script":"above testing.sh script here"}'
j = json.loads(json_script)
shell_script = j['script']
subprocess.call(shell_script, shell=True)

But everytime I am trying, it is giving me an error -

Below is my full Python script which contains the above testing.sh shell script in a JSON document -

#!/usr/bin/python

import subprocess
import json
import socket
import os

jsonData = '{"pp": [0,3,5,7,9], "sp": [1,2,4,6,8]}'
jj = json.loads(jsonData)

print jj['pp']
print jj['sp']

os.putenv( 'jj1',  'Hello World 1')
os.putenv( 'jj2',  'Hello World 2')
os.putenv( 'jj3', ' '.join( str(v) for v in jj['pp']  ) )
os.putenv( 'jj4', ' '.join( str(v) for v in jj['sp']  ) )

print "start"

jsonDataaa = '{"script":"#!/bin/bash \\n hello=$jj1 \\n echo $hello \\n echo $jj1 \\n echo $jj2 \\n for el1 in $jj3 \\n do \\n echo "$el1" \\n done \\n for el2 in $jj4 \\n do \\n echo "$el2" \\n done"}'
j = json.loads(jsonDataaa)

shell_script = j['script']
print "start"
subprocess.call(shell_script, shell=True)
print "end"

And below is the error I am getting -

 File "C:\Python27\lib\json\__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "C:\Python27\lib\json\decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python27\lib\json\decoder.py", line 381, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting , delimiter: line 1 column 113 (char 112) 

And the expected output I should be getting like this -

[0, 3, 5, 7, 9]
[1, 2, 4, 6, 8]
start
Hello World 1
Hello World 2
0
3
5
7
9
1
2
4
6
8
end

UPDATE:-

If I have my jsonDataaa like this

jsonDataaa = '{"script":"#!/bin/bash \\n hello=$jj1 \\n echo $hello \\n echo $jj1 \\n echo $jj2 \\n"}'

then it works fine... And I am able to execute it properly.. But if my jsonDataaa is as I mentioned in my question, then only it gives me an error. I am thinking there might be some syntax error which I am not able to understand.

Answer 1

It gives you this error because your json string is invalid. Specifically, it contains unescaped quotes.

It works if you replace your jsonDataaa assignment with:

jsonDataaa = '{"script":"#!/bin/bash \\n hello=$jj1 \\n echo $hello \\n echo $jj1 \\n echo $jj2 \\n for el1 in $jj3 \\n do \\n echo \\"$el1\\" \\n done \\n for el2 in $jj4 \\n do \\n echo \\"$el2\\" \\n done"}'

Answer 2

This part doesn't quite make sense:

jsonDataaa = '{"script":"#!/bin/bash \\n hello=$jj1 \\n echo $hello \\n echo $jj1 \\n echo $jj2 \\n for el1 in $jj3 \\n do \\n echo "$el1" \\n done \\n for el2 in $jj4 \\n do \\n echo "$el2" \\n done"}'
j = json.loads(jsonDataaa)

shell_script = j['script']
print "start"
subprocess.call(shell_script, shell=True)

What you've done here is pass a literal string as the argument to subprocess.call(), but it expects a program name, not program text. So you have a couple choices: you could write the contents of shell_script to a NamedTemporaryFile and execute that, or you could launch a piped bash in subprocess and feed its stdin from shell_script as a string. That latter approach is what I'd prefer, and you can get more help to do it here: Python - How do I pass a string into subprocess.Popen (using the stdin argument)?

PS: if you use a triple-quoted string you can make your JSON look a lot nicer, on multiple lines and whatnot.

Answer 3

You could also do:

subprocess.call(["/bin/sh", "-c", shell_script])

You are passing the whole script as a single command-line parameter to sh. This should work even if there are newlines in shell_script; but if it's more than a certain size (16K bytes? something like that) it won't fit in a command line.

EDIT - by using subprocess.call(shell_script, shell=True) it actually amounts to the same as what I've suggested, more or less (perhaps the effect of other options might be different). So now I'm guessing the quoting of the \\n etc may be incorrect as you've hinted at - see comments below.