running bash command from python shell

Question

I want to run a bash command from python shell. my bash is:

grep -Po "(?<=<cite>).*?(?=</cite>)" /tmp/file1.txt | awk -F/ '{print $1}' | awk '!x[$0]++' > /tmp/file2.txt

what I tried is:

#!/usr/bin/python
import commands
commands.getoutput('grep ' + '-Po ' +  '\"\(?<=<dev>\).*?\(?=</dev>\)\" ' + '/tmp/file.txt ' + '| ' + 'awk \'!x[$0]++\' ' + '> ' + '/tmp/file2.txt')

But I don't have any result.

Thank you

Answer 1

The recommend way to run system commands in python is to use the module subprocess .

import subprocess

a=['grep' ,'-Po', '"(?<=<dev>).*?(?=</dev>)"','/tmp/file.txt']
b=['awk', '-F/', '"{print $1}"'] 
c=["awk", '"!x[$0]++"']

p1 = subprocess.Popen(a,stdout=subprocess.PIPE)
p2 = subprocess.Popen(b,stdin=p1.stdout,stdout=subprocess.PIPE)
p3 = subprocess.Popen(c,stdin=p2.stdout,stdout=subprocess.PIPE)
p1.stdout.close()
p2.stdout.close()
out,err=p3.communicate()
print out

The point of creating pipes between each subprocess is for security and debugging reasons. Also it makes the code much clearer in terms, which process gets input and sends output to.

Answer 2

If you want to avoid splitting your arguments and worrying about pipes, you can use the shell=True option:

cmd = "grep -Po \"(?<=<dev>).*?(?=</dev>)\" /tmp/file.txt | awk -F/ '{print $1}' | awk '!x[$0]++' > file2.txt"
out = subprocess.check_output(cmd, shell=True)

This will run a subshell which will understands all your directives, including "|" for piping, ">" for redirection. If you do not do this, these symbols normally parsed by the shell will just be passed to grep program.

Otherwise, you have to create the pipes yourself. For example (untested code below):

grep_p = subprocess.Popen(["grep", "-Po", "(?<=<dev>).*?(?=</dev>)", "/tmp/file.txt"], stdout=subprocess.PIPE)
awk_p = subprocess.Popen(["awk", "-F/", "'{print $1}'"], stdin = grep_p.stdout)
file2_fh = open("file2.txt", "w")
awk_p_2 = subprocess.Popen(["awk", "!x[$0]++", stdout = file2_fh, stdin = awk_p.stdout)
awk_p_2.communicate()

However, you're missing the point of python if you are doing this. You should instead look into the re module: re.match , re.sub , re.search , though I'm not familiar enough with awk to translate your commands.

Answer 3

you must use

import os 
os.system(command)

Answer 4

Let us write a simple function to easily deal with these messy pipes for us:

def subprocess_pipes (pipes, last_pipe_out = None):
    import subprocess
    from subprocess  import PIPE
    last_p = None
    for cmd in pipes:
        out_pipe = PIPE if not (cmd==pipes[-1] and last_pipe_out) else open(last_pipe_out, "w")
        cmd = cmd if isinstance(cmd, list) else cmd.split(" ")
        in_pipe = last_p.stdout if last_p else None
        p = subprocess.Popen(cmd, stdout = out_pipe, stdin = in_pipe)
        last_p = p
    comm = last_p.communicate()
    return comm

Then we run,

subprocess_pipes(("ps ax", "grep python"), last_pipe_out = "test.out.2")

The result is a "test.out.2" file with the contents of piping "ps ax" into "grep python".

In your case,

a = ["grep", "-Po", "(?<=<cite>).*?(?=</cite>)", "/tmp/file1.txt"]
b = ["awk", "-F/", "{print $1}"]
c = ["awk", "!x[$0]++"]

subprocess_pipes((a, b, c),  last_pipe_out = "/tmp/file2.txt")

Answer 5

The commands module is obsolete now. If you don't actually need the output of your command you can use

import os

exit_status = os.system("your-command")

Otherwise you can use

import suproccess

out, err = subprocess.Popen("your | commands", stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell = True).communicate()

Note: for your command you send stdout to file2.txt so I wouldn't expect to see anything in out you will however still see error messages on stderr which will go into err

Answer 6

I think what you are looking for is something like: ubprocess.check_output(same as popen arguments, **kwargs) , use it the same way you would use a popen command , it should show you the output of the program that's being called.

For more details here is a link: http://freefilesdl.com/how-to-call-a-shell-command-from-python/