Python爬行Pastebin（JavaScript呈现的网页）

Question

I am facing a problem trying to crawl JavaScript rendered pages.

I am using the python-qt4 module, following this tutorial: https://impythonist.wordpress.com/2015/01/06/ultimate-guide-for-scraping-javascript-rendered-web-pages/

In the tutorial, everything works perfectly with the example page: http://pycoders.com/archive

But I am trying this out with pastebin, with this URL:

http://pastebin.com/search?q=ssh

What I am trying is to get all the links, in order to click them, as well as be able to follow the pages (I don't know what I am going to use yet, maybe Scrapy but I want to take a look to other options).

The problem is that I am not able to extract the links, this is my code:

import sys  
from PyQt4.QtGui import *  
from PyQt4.QtCore import *  
from PyQt4.QtWebKit import *  
from lxml import html 

#Take this class for granted.Just use result of rendering.
class Render(QWebPage):  
  def __init__(self, url):  
    self.app = QApplication(sys.argv)  
    QWebPage.__init__(self)  
    self.loadFinished.connect(self._loadFinished)  
    self.mainFrame().load(QUrl(url))  
    self.app.exec_()  

  def _loadFinished(self, result):  
    self.frame = self.mainFrame()  
    self.app.quit()  

url = 'http://pastebin.com/search?q=ssh'  
r = Render(url)  
result = r.frame.toHtml()
formatted_result = str(result.toAscii())
tree = html.fromstring(formatted_result)
archive_links = tree.xpath('//a[@class="gs-title"]/@data-ctoring')
for i in archive_links:
    print i

The result is: I don't get anything.

Answer 1

Ideally, you should look into using Pastebin API - here is a Python wrapper .

Alternative approach would involve browser automation via selenium . Working code that prints the search result links:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Firefox()
driver.get("http://pastebin.com/search?q=ssh")

# wait for the search results to be loaded
wait = WebDriverWait(driver, 10)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".gsc-result-info")))

# get all search results links
for link in driver.find_elements_by_css_selector(".gsc-results .gsc-result a.gs-title"):
    print(link.get_attribute("href"))