使用lxml和Xpath刮取元素

Question

The issue I'm having is scraping out the element itself. I'm able to scrape the first two (IncidentNbr and DispatchTime ) but I can't get the address... (1300 Dunn Ave) I want to be able to scrape that element but also have it dynamic enough so I'm not actually parsing for "1300 Dunn Ave" I'm parsing for that element. Here is the source code

<td><span id="lstCallsForService_ctrl0_lblIncidentNbr">150318182198</span></td>
<td><nobr><span id="lstCallsForService_ctrl0_lblDispatchTime">3-18 10:25</span></nobr></td>
<td>
    <a id="lstCallsForService_ctrl0_lnkAddress" href="https://maps.google.com/?q=1300 DUNN AVE, Jacksonville, FL" target="_blank" style="text-decoration:underline;">1300 DUNN AVE</a>
</td>

And here is my code:

from lxml import html
import requests

page = requests.get('http://callsforservice.jaxsheriff.org/')
tree = html.fromstring(page.text)

callSignal = tree.xpath('//span[@id="lstCallsForService_ctrl0_lblIncidentNbr"]/text()')
dispatchTime = tree.xpath('//span[@id="lstCallsForService_ctrl0_lblDispatchTime"]/text()')
location = tree.xpath('//span[@id="lstCallsForService_ctrl0_lnkAddress"]/text()')



print 'Call Signal: ', callSignal
print "Dispatch Time: ", dispatchTime
print "Location: ", location

And this is my output:

Call Signal:  ['150318182198']
Dispatch Time:  ['3-18 10:25']
Location:  []

Any idea on how I can scrape out the address?

Answer 1

This is the element you are looking for:

<a id="lstCallsForService_ctrl0_lnkAddress"
   href="https://maps.google.com/?q=1300 DUNN AVE, Jacksonville, FL"
   target="_blank" style="text-decoration:underline;">1300 DUNN AVE</a>

As you can see, it is not a span element. Your current XPath expression:

//span[@id="lstCallsForService_ctrl0_lnkAddress"]/text()

is looking for a span element with this ID, when it should actually be selecting an a element. Use

//a[@id="lstCallsForService_ctrl0_lnkAddress"]/text()

instead. Then, the result should be

Location:  ['1300 DUNN AVE']

Please also read alecxe's answer which has more practical advice than mine.

Answer 2

First of all, it is an a element, not a span . And you need a double slash before the text() :

//a[@id="lstCallsForService_ctrl0_lnkAddress"]//text()

Why a double slash? This is because in reality this a element has no direct text node children:

<a id="lstCallsForService_ctrl0_lnkAddress" href="https://maps.google.com/?q=5100 CLEVELAND RD, Jacksonville, FL" target="_blank">
    <u>5100 CLEVELAND RD</u>
</a>

You could also reach the text through u tag:

//a[@id="lstCallsForService_ctrl0_lnkAddress"]/u/text()

---

Besides, to scale the solution into multiple results:

• iterate over table rows
• for every row find the cell values using a partial id attribute match using contains()
• use text_content() method to get the text

Implementation:

for item in tree.xpath('//tr[@class="closedCall"]'):
    callSignal = item.xpath('.//span[contains(@id, "lblIncidentNbr")]')[0].text_content()
    dispatchTime = item.xpath('.//span[contains(@id, "lblDispatchTime")]')[0].text_content()
    location = item.xpath('.//a[contains(@id, "lnkAddress")]')[0].text_content()

    print 'Call Signal: ', callSignal
    print "Dispatch Time: ", dispatchTime
    print "Location: ", location
    print "------"

Prints:

Call Signal:  150318182333
Dispatch Time:  3-18 11:22
Location:  9600 APPLECROSS RD
------
Call Signal:  150318182263
Dispatch Time:  3-18 11:12
Location:  1100 E 1ST ST
------
...