我如何使用lxml xpath从Web抓取数据中获取特定元素

Question

I followed the link below to scrape historical data from Prize Zombie:

https://impythonist.wordpress.com/2015/01/06/ultimate-guide-for-scraping-javascript-rendered-web-pages/

The script I have is like below:

import requests  
import pandas as pd
import sys    
import csv  
import urllib2  
import sys  
import time  
from bs4 import BeautifulSoup  
from PyQt4.QtGui import *  
from PyQt4.QtCore import *  
from PyQt4.QtWebKit import *  
from lxml import html 

class Render(QWebPage):  
  def __init__(self, url):  
    self.app = QApplication(sys.argv)  
    QWebPage.__init__(self)  
    self.loadFinished.connect(self._loadFinished)  
    self.mainFrame().load(QUrl(url))  
    self.app.exec_()  

  def _loadFinished(self, result):  
    self.frame = self.mainFrame()  
    self.app.quit() 

url = 'https://www.pricezombie.com/viewproduct/pF/5jNvj/Align-Probiotic-Supplement-42-count'

r = Render(url)  

result = r.frame.toHtml()

formatted_result = str(result.toAscii())

tree = html.fromstring(formatted_result)

According the author, now I need to use xpath to get the element I want.
However, I really can't figure out how to get those specific elements from the tree.

The html part should look like this:

class="pt1">$27.51, May 15 - Jun 10   

And the information I need is:

<g class="pzmo">
    <rect x="91" y="14" height="216" width="7" style="fill:#ccc" fill-opacity="0.2"></rect>
    <rect fill-opacity="0.9" class="prec" x="98" y="14" width="170" height="20"></rect>
    <text x="103" y="28" class="pt1">$27.51, May 15 - Jun 10</text>
</g>

Could anyone tell me what's the xpath for that?

Answer 1

The xpath is, for example:

//*[@id="chart3Dqt"]/svg/g[412]/text[1]

If you want to vary the index you can substitute with a formatting field:

>>> xpath = '//*[@id="chart3Dqt"]/svg/g[{index}]/text[1]'

>>> xpath.format(index=412)
//*[@id="chart3Dqt"]/svg/g[412]/text[1]