OSError：[Errno 36]使用Popen时文件名太长-Python

Question

As I started asking on a previous question, I'm extracting a tarball using the tarfile module of python. I don't want the extracted files to be written on the disk, but rather get piped directly to another program, specifically bgzip.

#!/usr/bin/env python
import tarfile, subprocess, re
mov = []
def clean(s):
   s = re.sub('[^0-9a-zA-Z_]', '', s)
   s = re.sub('^[^a-zA-Z_]+', '', s)
   return s
with tarfile.open("SomeTarballHere.tar.gz", "r:gz") as tar:
    for file in tar.getmembers():
        if file.isreg():
            mov = file.name
            proc = subprocess.Popen(tar.extractfile(file).read(), stdout = subprocess.PIPE)
            proc2 = subprocess.Popen('bgzip -c > ' + clean(mov), stdin = proc, stdout = subprocess.PIPE)           
            mov = None

But now I get stuck on this:

Traceback (most recent call last):
  File "preformat.py", line 12, in <module>
    proc = subprocess.Popen(tar.extractfile(file).read(), stdout = subprocess.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 36] File name too long

Is there any workaround for this? I have been using the LightTableLinux.tar.gz (it contains the files for a text editor program) as a tarball to test the script on it.

Answer 1

The exception is raised in the forked-off child process when trying to execute the target program from this invocation:

proc = subprocess.Popen(tar.extractfile(file).read(), stdout = subprocess.PIPE)

This

1. reads the contents of an entry in the tar file
2. tries to execute a program with the name of the contents of that entry.

Also your second invocation won't work, as you are trying to use shell redirection without using shell=True in Popen() :

proc2 = subprocess.Popen('bgzip -c > ' + clean(mov), stdin = proc, stdout = subprocess.PIPE) 

The redirect may also not be necessary, as you should be able to simply redirect the output from bgzip to a file from python directly.

Edit: Unfortunately, despite extractfile() returning a file-like object , Popen() expects a real file (with a fileno ). Hence, a little wrapping is required:

with tar.extractfile(file) as tarfile, file(clean(mov), 'wb') as outfile:
    proc = subprocess.Popen(
        ('bgzip', '-c'),
        stdin=subprocess.PIPE,
        stdout=outfile,
    )
    shutil.copyfileobj(tarfile, proc.stdin)
    proc.stdin.close()
    proc.wait()