reproduce 'du' command results in python

Question

I am trying to reproduce du command using python. The problem is that sometimes I don't have access to some files / directories, which I can usually skip by redirecting to null or hiding the Permission denied lines by using grep -v

Here is the function

def du(path):
    """disk usage in kilobytes"""
    print "calculating disk usage for " + path + " ..."
    # return subprocess.check_output(['du', '-s',
    # path]).split()[0].decode('utf-8')
    try:
        output = subprocess.check_output(['ls', '-d', path, '|', 'parallel', '--no-notice', 'du', '-s', '2>&1', '|', 'grep', '-v', '"Permission denied"'], shell=True, stderr=subprocess.STDOUT).split()[0].decode('utf-8')
    except subprocess.CalledProcessError as e:
        raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
    return output

The problem is that it captures the exit code and throw an error anyway, is there something I can change in this function to skip the permission denied lines ?

Thanks

EDIT

I added a modification to the function that worked for me in case anyone wants to get this done one day here is the updated function

def du(path):
    """disk usage in kilobytes"""
    print "calculating disk usage for " + path + " ..."
    # return subprocess.check_output(['du', '-s',
    # path]).split()[0].decode('utf-8')
    try:
        p1 = subprocess.Popen(('ls', '-d', path), stdout=subprocess.PIPE)
        p2 = subprocess.Popen(('parallel', '--no-notice', 'du', '-s', '2>&1'), stdin=p1.stdout, stdout=subprocess.PIPE)
        p3 = subprocess.Popen(('grep', '-v', '"Permission denied"'), stdin=p2.stdout, stdout=subprocess.PIPE )
        output = p3.communicate()[0]
        #output = subprocess.check_output(['ls', '-d', path, '|', 'parallel', '--no-notice','du', '-s', '2>&1', '|', 'grep', '-v', '"Permission denied"'], shell=True, stderr=subprocess.STDOUT).split()[0].decode('utf-8')
    except subprocess.CalledProcessError as e:
        raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
    return ''.join([' '.join(hit.split('\t')) for hit in output.split('\n') if len(hit) > 0 and not "Permission" in hit])

Answer 1

If you do not want it to throw the error, do not use raise in your except block. Maybe just print to stderr. You may also want to be more complete and check the particular error to make sure you only avoid throwing specific ones.

Also, you may want to look into using os.path.getsize instead of shelling out to ls.

Answer 2

Perhaps you can just look the built-in du of the sh module.

from sh import du

You can then call it with the arguments you'd like:

du()

Everything you need on the module can be found here: https://github.com/amoffat/sh (including installation, and including a discussion of sudo ).

Quoth:

sh (previously pbs) is a full-fledged subprocess replacement for Python 2.6 - 3.4 that allows you to call any program as if it were a function...