python subprocess.Popen vs os.popen

Question

I'm trying to get the output of the following shell command in my python script,

hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}

I can successfully get the output through os.popen as follows:

import os
cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
p = os.popen(cmd,"r")
while 1:
    line = p.readline()
    if not line: break
    print line

But os.popen() is deprecated since python 2.6 so I wanted to replace the above snippet with the subprocess.Popen() function.

But the code snippet for subprocess.Popen() below gives a different result than the code snippet above.

import subprocess as sub
import shlex
cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
args = shlex.split(cmd)
p = sub.Popen(args,stdout=sub.PIPE,stderr=sub.PIPE)
output, errors = p.communicate()
print output

The above command just gives output for 'hadoop fs -ls /projectpath/' part of the command. I have tried consulting several references ( http://docs.python.org/2/library/subprocess.html#popen-objects , Python, os.system for command-line call (linux) not returning what it should? ) for subpocess.Popen() but cannot get it to execute the command in the string cmd. Can anyone point out what I'm doing wrong?

Answer 1

try this:

cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
p = sub.Popen(cmd,stdout=sub.PIPE,stderr=sub.PIPE, shell=True)