Any way to execute a piped command in Python using subprocess module, without using shell=True?

Question

I want to run a piped command line linux/bash command from Python, which first tars files, and then splits the tar file. The command would look like something this in bash:

> tar -cvf - path_to_archive/* | split -b 20m -d -a 5 - "archive.tar.split"

I know that I could execute it using subprocess, by settings shell=True, and submitting the whole command as a string, like so:

import subprocess    

subprocess.call("tar -cvf - path_to_archive/* | split -b 20m -d -a 5 - 'archive.tar.split'", shell=True)

...but for security reasons I would like to find a way to skip the "shell=True" part, (which takes a list of strings rather than a full command line string, and which can not handle the pipe char correctly). Is there any solution for this in Python? Ie, is it possible to set up linked pipes somehow, or some other solution?

Answer 1

If you want to avoid using shell=True, you can manually use subprocess pipes .

from subprocess import Popen, PIPE
p1 = Popen(["tar", "-cvf", "-", "path_to_archive"], stdout=PIPE)
p2 = Popen(["split", "-b", "20m", "-d", "-a", "5", "-", "'archive.tar.split'"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

Note that if you do not use the shell, you will not have access to expansion of globbing characters like *. Instead you can use the glob module.

Answer 2

tar can split itself:

tar -L 1000000 -F name-script.sh cf split.tar largefile1 largefile2 ...

name-script.sh

#!/bin/bash
echo "${TAR_ARCHIVE/_part*.tar/}"_part"${TAR_VOLUME}".tar >&"${TAR_FD}"

To re-assemble

tar -M -F name-script.sh cf split.tar

Add this to your python program.

Answer 3

Is there any reason you can't use tarfile? | http://docs.python.org/library/tarfile.html

import tarfile
tar = tarfile.open("sample.tar.gz")
tar.extractall()
tar.close()

Just write like a file like object using tarfile rather than invoking subprocess.

Answer 4

Shameless plug, I wrote a subprocess wrapper for easier command piping in python: https://github.com/houqp/shell.py

Example:

shell.ex("tar -cvf - path_to_archive") | "split -b 20m -d -a 5 - 'archive.tar.split'"