Python script : Running a script with multiple arguments using subprocess

Question

My question is related to this earlier question - Python subprocess usage

I am trying to run this command using python

nccopy -k 4 " http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40] " foo.nc

When I run the above command I should be able to see a file called foo.nc on my disk or a network error stating unable to access that URL or remote URL not found.

Currently the ESRL NOAA server is down - so when I run the above command I get

syntax error, unexpected $end, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR context: ^ NetCDF: Access failure Location: file nccopy.c; line 1348

I should get the same error when I run the python script

This is the code I have and I am unable to figure out exactly how to proceed further -

I tried splitting up "-k 4" into two arguments and removing the quotes and I still get this error nccopy : invalid format : 4

Results of print(sys.argv) data.py

['data.py', '-k', '4', ' http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][20:34][26:40] ', 'foo.nc']

import numpy as np

import subprocess

import sys

url = '"http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]"'

outputFile = 'foo.nc'

arg1 = "-k 4"

arg3 = url 

arg4 = outputFile


print (input)

subprocess.check_call(["nccopy",arg1,arg3,arg4])

Answer 1

Instead of arg1 = "-k 4", use two arguments instead.

import subprocess


url = 'http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]'

outputFile = 'foo.nc'

arg1 = "-k"
arg2 = "4"
arg3 = url 
arg4 = outputFile

subprocess.check_call(["nccopy", arg1, arg2, arg3, arg4])

See also here Python subprocess arguments

Answer 2

There's two dilemmas here.
One being that subprocess processes your arguments and tries to use 4 as a separate argument.

The other being that system calls still goes under normal shell rules, meaning that parameters and commands will be parsed for metacharacters aka special characters. In this case you're wrapping [ and ] .

There for you need to separate each parameters and it's value into separate objects in the parameter-list, for instance -k 4 should be ['-k', '4'] and you need to wrap parameters/values in '...' instead of "..." .

Try this, shlex.split() does the grunt work for you, and i swapped the encapsulation characters around the URL:

import numpy as np
import subprocess
import sys
import shlex

url = "'http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]'"

outputFile = 'foo.nc'
command_list = shlex.split('nccopy -k 4 ' + url + ' ' + outpufFile)

print(command_list)

subprocess.check_call(command_list)

Answer 3

If you have a working shell command that runs a single program with multiple arguments and you want to parameterized it eg, to use a variable filename instead of the hardcoded value then you could use shlex.split() to create a list of command-line arguments that you could pass to subprocess module and replace the desired argument with a variable eg:

>>> shell_command = "python -c 'import sys; print(sys.argv)' 1 't w o'"
>>> import shlex
>>> shlex.split(shell_command)
['python', '-c', 'import sys; print(sys.argv)', '1', 't w o']

To run the command using the same Python interpreter as the parent script, sys.executable could be used and we can pass a variable instead of '1' :

#!/usr/bin/env python
import random
import sys
import subprocess

variable = random.choice('ab')
subprocess.check_call([sys.executable, '-c', 'import sys; print(sys.argv)',
                       variable, 't w o'])

Note:

• one command-line argument per list item
• no shlex.split() in the final code
• there are no quotes inside 'tw o' ie, 'tw o' is used instead of '"two"' or "'tw o'"

subprocess module does not run the shell by default and therefore you don't need to escape shell meta-characters such as a space inside the command-line arguments. And in reverse, if your command uses some shell functionality (eg, file patterns) then either reimplement the corresponding features in Python (eg, using glob module) or use shell=True and pass the command as a string as is. You might need pipes.quote() , to escape variable arguments in this case. Wildcard not working in subprocess call using shlex