Run shell script in python

Question

I need to execute a shell script to run the python program in via python command.

I should have to execute my python script like this

ubuntu@ip-10-32-157-231:~/hg_intcen/lib$ xvfb-run python webpage_scrapper.py  http://www.google.ca/search?q=navaspot

This script need to be executed in python program since there are huge links has to be passed to that module.

I have searched to execute this shell script in python,so i used "subprocess"

The main thing is when you run this shell command it takes some time to return the result. i need the python module to execute this command as well as it has to wait for while to return the result.This is required.

I used subprocess.Popen it doesn't return the result like what i got from the bash

import subprocess
def execute_scrapping(url):
   exe_cmd = "xvfb-run python lib/webpage_scrapper.py"+" "+str(url)
   print "cmd:"+exe_cmd
   proc = subprocess.Popen(exe_cmd,shell=True,stdin=subprocess.PIPE,stdout=subprocess.PIPE)
   time.sleep(15)
   sys.stdout.flush()
   d=proc.stdout.readlines()
   return d[1]

this above is not run into exact result. Could you please suggest me to execute the bash shell command via python and get the result?

Answer 1

Try:

proc.wait()

instead of your time.sleep(15) call.

From the docs:

Popen.wait() - Wait for child process to terminate. Set and return returncode attribute.

Answer 2

You should use the communicate() method to wait that the external process completes.

stddata, stderr = proc.communicate()

If you have to exchange messages between the two process then look into the pexpect module:

From the website:

   import pexpect
   child = pexpect.spawn ('ftp ftp.openbsd.org')
   child.expect ('Name .*: ')
   child.sendline ('anonymous')
   child.expect ('Password:')
   child.sendline ('noah@example.com')
   child.expect ('ftp> ')
   child.sendline ('cd pub')
   child.expect('ftp> ')
   child.sendline ('get ls-lR.gz')
   child.expect('ftp> ')
   child.sendline ('bye')