在Django中使用QT刮取一次，在main（）线程中未创建使用QApplication的下一次运行时崩溃

Question

I am working on creating a Django-based scraper in which a user can enter a search term. I use that search term(s) to build a URL and query the site, then returning un-rendered HTML and JS. I am then able to take the post request, render the page by creating a Qwebpage, passing it the URL and grabbing the frame's rendered HTML. This works one time in my Django app, and the next POST request crashes the site.

My first concern is that in this current set up, I am forced to use the xvfb-run wrapper to run. Is this going to pose an issue when I deploy - better question is: can I use an xvfb wrapper in production somehow?

With that said I am able to make one post request and this returns the page that I am looking for. If I hit back, and send another request, I see the following errors in console, and this then shuts down the ./manage.py server:

WARNING: QApplication was not created in the main() thread.
QObject::connect: Cannot connect (null)::configurationAdded(QNetworkConfiguration) to QNetworkConfigurationManager::configurationAdded(QNetworkConfiguration)
QObject::connect: Cannot connect (null)::configurationRemoved(QNetworkConfiguration) to QNetworkConfigurationManager::configurationRemoved(QNetworkConfiguration)
QObject::connect: Cannot connect (null)::configurationChanged(QNetworkConfiguration) to QNetworkConfigurationManager::configurationChanged(QNetworkConfiguration)
QObject::connect: Cannot connect (null)::onlineStateChanged(bool) to QNetworkConfigurationManager::onlineStateChanged(bool)
QObject::connect: Cannot connect (null)::configurationUpdateComplete() to QNetworkConfigurationManager::updateCompleted()
Segmentation fault (core dumped)

I will admit that I do not understand what in particular the error is here since I'm rather new to threading concepts. I am uncertain if this error means that it can't reconnect to the xvfb wrapper thats already running, or if indeed it is a threading issue. The code that works once is here. This has been changed slightly since I don't want to show the site I'm actually scraping. Also, I am not hunting for data in this sample. This sample will simply bring rendered HTML to your browser as a test:

import sys
from django.shortcuts import render

# Create your views here.
from django.http import HttpResponse
from django.http import HttpResponseRedirect
from django.views.generic import View
from PyQt4.QtGui import *  
from PyQt4.QtCore import *  
from PyQt4.QtWebKit import * 
from bs4 import BeautifulSoup 

from .forms import QueryForm

def query(request):
        results = google.search("Real Estate")
        context = {'results': results}
        return render(request, 'searchlistings/search.html', context)

class Render(QWebPage):  
  def __init__(self, url):  
    self.app = QApplication(sys.argv)  
    QWebPage.__init__(self)  
    self.loadFinished.connect(self._loadFinished)  
    self.mainFrame().load(QUrl(url))  
    self.app.exec_()  

  def _loadFinished(self, result):  
    self.frame = self.mainFrame()  
    self.app.quit()

class SearchView(View):
    form_class = QueryForm
    template_name = 'searchlistings/index.html'

    def get(self, request, *args, **kwargs):
        form = self.form_class()
        return render(request, self.template_name, {'form': form})

    def post(self, request, *args, **kwargs):
        form = self.form_class(request.POST)
        if form.is_valid():
            query = form.cleaned_data['query']
            context = self.isOnSite(query)
            #return context
            #return render(request, 'searchlistings/search.html', {'context': context})
            return HttpResponse(context)

    def isOnSite(self, query):
        url = "http://google.com"
        #This does the magic.Loads everything
        r = Render(url)  
        #result is a QString.
        result = r.frame.toHtml()
        r.app.quit()
        return result;

So my primary questions are this:

1. Is XVFB wrapper appropriate here and can I use this set up in production on a different host. Will this work not on my local vagrant box?

2. The main() thread issue - is this a threading issue or an issue not connecting back to the xvfb server? Can this issue be resolved with Celery or something similar?

3. Is this an appropriate way to do what I want? I've seen lots of other solutions including scrapyjs, spynner, selenium and so on but they seem either overtly complicated or based on QT. A better question is do any of these alternative packages solve the main() thread issue?

Thanks for your help!

Answer 1

OK the solution here was to use twill as documented here http://twill.idyll.org/python-api.html - I am able to run this without the xvfb wrapper and it is much faster than previous methods with much less overhead. I can recommend this.