Calling bash command inside Python returns error, but works in terminal

Question

Here is the except of my code related to this:

def grd_commands(directory):
    for filename in os.listdir(directory)[1:]:
        print filename
        new_filename = ''
        first_letter = ''
        second_letter = ''
        bash_command = 'gmt grdinfo ' + filename + ' -I-'
        print bash_command
        coordinates = Popen(bash_command, stdout=PIPE, shell=True)
        coordinates = coordinates.communicate()
        latlong = re.findall(r'^\D*?([-+]?\d+)\D*?[-+]?\d+\D*?([-+]?\d+)', coordinates)
        if '-' in latlong[1]:
            first_letter = 'S'
        else:
            first_letter = 'N'
        if '-' in latlong[0]:
            second_letter = 'W'
        else:
            second_letter = 'E'

        new_filename = first_letter + str(latlong[1]) + second_letter + str(latlong[0]) + '.grd'
        Popen('gmt grdconvert ' + str(filename) + ' ' + new_filename, shell=True)

filename is the name of the file that is is being passed to the function. When I run my code, I am receiving this error:

/bin/sh: gmt: command not found
Traceback (most recent call last):
  File "/Users/student/Desktop/Code/grd_commands.py", line 38, in <module>
    main()
  File "/Users/student/Desktop/Code/grd_commands.py", line 10, in main
    grd_commands(directory)
  File "/Users/student/Desktop/Code/grd_commands.py", line 23, in grd_commands
    latlong = re.findall(r'^\D*?([-+]?\d+)\D*?[-+]?\d+\D*?([-+]?\d+)', coordinates).split('\n')
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/re.py", line 177, in findall
    return _compile(pattern, flags).findall(string)
TypeError: expected string or buffer

If I print out the string bash_command and try entering it into terminal it fully functions. Why doesn't it work when being called by my Python script?

Answer 1

The entire command line is being treated as a single command name . You need to either use shell=True to have the shell parse it as a command line:

coordinates = Popen(bash_command, stdout=PIPE, shell=True)

or preferably store the command name and its arguments as separate elements of a list:

bash_command = ['gmt', 'grdinfo', filename, '-I-']
coordinates = Popen(bash_command, stdout=PIPE)

Answer 2

Popen takes a list of arguments. There is a warning for using shell=True

Passing shell=True can be a security hazard if combined with untrusted input.

Try this:

from subprocess import Popen, PIPE
bash_command = 'gmt grdinfo ' + filename + ' -I-'
print(bash_command)
coordinates = Popen(bash_command.split(), stdout=PIPE)
print(coordinates.communicate()[0])

Ensure gmt is installed in a location specified by PATH in your /etc/environment file:

PATH=$PATH:/path/to/gmt

Alternatively, specify the path to gmt in bash_command:

bash_command = '/path/to/gmt grdinfo ' + filename + ' -I-'

You should be able to find the path with:

which gmt

As other people have suggested, an actual list would be the best approach instead of a string. Additionally, you must escape spaces with a '\\' in order to actually access the file if there is a space in it.

for filename in os.listdir(directory)[1:]:
    bash_command = ['gmt', 'grdinfo', filename.replace(" ", "\ "), '-I-']