I followed the link below to scrape historical data from Prize Zombie:
The script I have is like below:
import requests
import pandas as pd
import sys
import csv
import urllib2
import sys
import time
from bs4 import BeautifulSoup
from PyQt4.QtGui import *
from PyQt4.QtCore import *
from PyQt4.QtWebKit import *
from lxml import html
class Render(QWebPage):
def __init__(self, url):
self.app = QApplication(sys.argv)
QWebPage.__init__(self)
self.loadFinished.connect(self._loadFinished)
self.mainFrame().load(QUrl(url))
self.app.exec_()
def _loadFinished(self, result):
self.frame = self.mainFrame()
self.app.quit()
url = 'https://www.pricezombie.com/viewproduct/pF/5jNvj/Align-Probiotic-Supplement-42-count'
r = Render(url)
result = r.frame.toHtml()
formatted_result = str(result.toAscii())
tree = html.fromstring(formatted_result)
According the author, now I need to use xpath to get the element I want.
However, I really can't figure out how to get those specific elements from the tree.
The html part should look like this:
class="pt1">$27.51, May 15 - Jun 10
And the information I need is:
<g class="pzmo">
<rect x="91" y="14" height="216" width="7" style="fill:#ccc" fill-opacity="0.2"></rect>
<rect fill-opacity="0.9" class="prec" x="98" y="14" width="170" height="20"></rect>
<text x="103" y="28" class="pt1">$27.51, May 15 - Jun 10</text>
</g>
Could anyone tell me what's the xpath for that?
The xpath is, for example:
//*[@id="chart3Dqt"]/svg/g[412]/text[1]
If you want to vary the index you can substitute with a formatting field:
>>> xpath = '//*[@id="chart3Dqt"]/svg/g[{index}]/text[1]'
>>> xpath.format(index=412)
//*[@id="chart3Dqt"]/svg/g[412]/text[1]
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.