Executing python script using laravel

Question

I am using Laravel console command to execute a python script.

This is what I have in Laravel console command file:

use Symfony\Component\Process\Process;
use Symfony\Component\Process\Exception\ProcessFailedException;

public function handle()
{
   $process = new Process(['python', 'App/Console/Python/Main.py']);

    $process->run();

    // executes after the command finishes
    if (!$process->isSuccessful()) {
        throw new ProcessFailedException($process);
    }

    $this->line($process->getOutput());
}

This is what I have in Main.py:

import YellScraper

scraper = YellScraper.Scraper()
scraper.setKeyword('Restaurant')
scraper.setLocation('Burnley')
scraper.start()
print('Done... Exiting')

This is what I have in YellScraper.py:

import urllib
import time
import sys
from bs4 import BeautifulSoup

class Scraper:
        pages = []
        keyword = ""
        location = ""
        results = []

        def setKeyword(self,keywordInput):
                self.keyword = keywordInput

        def setLocation(self,locationInput):
                self.location = locationInput

        def createPages(self):
                if self.keyword != "" and self.location != "":
                        for i in range(1,10):
                                page = 'http://www.yell.com/ucs/UcsSearchAction.do?keywords='+self.keyword+'&location='+self.location+'&pageNum=%s' % i
                                self.pages.append(page)
                else:
                        print ("Location or Keyword not set\nUse setKeyword or setLocation functions before continuing")
                        sys.exit(0)

        def start(self):
                self.createPages()
                if self.pages:
                        for page in self.pages:
                                f = urllib.urlopen(page)
                                f = f.read()
                                soup = BeautifulSoup(f)
                                listings = soup.find_all("div", class_="parentListing")
                                if len(listings) > 0:
                                    for listing in listings:
                                        company = {}

                                        #name
                                        company['name'] = listing.find('a', itemprop="name").get_text(strip=True)

                                        #number
                                        company['number'] = listing.select('.l-telephone ul li strong')[0].string

                                        #address
                                        company['address'] = listing.find('span', itemprop="streetAddress").get_text(strip=True) + ' ' + listing.find('span', itemprop="addressLocality").get_text(strip=True) + ', ' + listing.find('span', itemprop="postalCode").get_text(strip=True)

                                        #add company
                                        self.results.append(company)

                                #keep yell.com happy
                                time.sleep(15)
                        return self.results

Whenever I execute this, I am getting an error saying maximum execution time 60 seconds. I believe Main.py doesn't know what is YellScraper, however not sure how to get this working

Answer 1

It seems like you want to run this python script on schedule and get an update back on the laravel application after it complete.

In this case, rather than running python script from laravel, I would set up a cronjob/window schedule the python script.

After the python script is complete, trigger an API call to the laravel app to update the necessary information so that your application will be decoupled and not rely on other app's executing time range or the library and will only communicate via API interface.