Why does tar fail to create an archive when called via Python's subprocess.call while using a * wildcard?

Question

I'm writing a python script that should extract some data from within the existing files and then pack the original files into several tarballs. The filename format looks like this:

system_yymmddT0000.zip

where system can be one of several names and YYMMDDThhmm is a date and time of creation.

To make this work, I'm using tar through Python's subprocess.call so, for files starting with date 1704 like SAP_1704T0000.zip , the command is:

subprocess.call(["tar", "-cvf", "SAP_2017_04.tar", "SAP_1704*", "1>", "SAP_2017_04.filelist"])

However, when I run this script, I get the following error:

tar: SAP_1704*: Cannot stat: No such file or directory
tar: 1>: Cannot stat: No such file or directory
tar: SAP_2017_04.filelist: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors

I also tried to pack all the arguments together like this:

subprocess.call(["tar", "-cvf SAP_2017_04.tar SAP_1704* 1> SAP_2017_04.filelist"]) (no commas between arguments). However, then I got the following error:

tar: Cowardly refusing to create an empty archive
Try `tar --help' or `tar --usage' for more information.

I can't figure out what I'm doing wrong, since manually navigating inside the folder and running the command tar cvf SAP_2017_04.tar SAP_1704* 1> SAP_2017_04.filelist works just fine.

Answer 1

Wildcards aren't processed by tar , they need to be processed by the program that calls it. Usually , that program is a shell.

However, it doesn't have to be; you can get much safer operation (if any of your parameters are user-configurable) by doing the work in native Python instead of using shell=True :

subprocess.call(['tar', '-cvf', 'SAP_2017_04.tar'] + glob.glob('SAP_1704*'),
                stdout=open('SAP_2017_04.filelist', 'w'))

• Instead of 1>somefile (an instruction to your shell to redirect stdout, FD 1, to write to somefile ), we use stdout=open('somefile', 'w') to tell Python the same thing.
• Instead of just putting SAP_1704* directly in the command line, we call glob.glob('SAP_1704*') in Python, and add the list it returns to the argument list.

Answer 2

Never mind, figured it out myself (at least I think so). It works like this:

substring = r"tar, -cvf SAP_2017_04.tar SAP_1704* 1> SAP_2017_04.filelist"
subprocess.call(substring, shell=True)

It seems there were two problems - escaping some special characters and also attempting to pass a list of arguments to the subprocess.call while it should be passed as a single string when using shell=True .

Big thanks to @CMMCD @BoarGules and @ErHarshRathore for putting me on the right track!