pipe/stream gnupg output to tarfile in

Question

I have the following code but obviously this is not real streaming. It is the best I could find but it reads the whole input file into memory first. I want to stream it to tarfile module without using all my memory when decrypting huge (>100Gb files)

import tarfile, gnupg                                                                                                                                                                                                                                
gpg = gnupg.GPG(gnupghome='C:/Users/niels/.gnupg')                                                                         

with open('103330-013.tar.gpg', 'r') as input_file:                                                                                                                                                                                                   
    decrypted_data = gpg.decrypt(input_file.read(), passphrase='aaa')                                                       
    # decrypted_data.data contains the data                                                                                 
    decrypted_stream = io.BytesIO(decrypted_data.data)                                                                      

    tar = tarfile.open(decrypted_stream, mode='r|')                                                                                                                                                                                                 
    tar.extractall()                                                                                                                                                                                                                                
    tar.close()

Answer 1

Apparently, you cannot use real streaming using gpnupg module, gnupg module always reads whole output of gnupg into memory. So to use real streaming, you'll have to run gpg program directly. Here is a sample code (without proper error handling):

import subprocess
import tarfile

with open('103330-013.tar.gpg', 'r') as input_file:
   gpg = subprocess.Popen(("gpg", "--decrypt", "--homedir", 'C:/Users/niels/.gnupg', '--passphrase', 'aaa'), stdin=input_file, stdout=subprocess.PIPE)
   tar = tarfile.open(fileobj=gpg.stdout, mode="r|")
   tar.extractall()
   tar.close()