[英]How to scrape html tags with no classes and changing id using scrapy?
[英]How to exclude a particular html tag(without any id) from several tags while using scrapy?
<div class="region size2of3">
<h2>Mumbai</h2>
<strong>Fort</strong>
<div>Elphinstone building, Horniman Circle,</div>
<div>Veer Nariman Road, Fort</div>
<div>Mumbai 400001</div>
<div>Timings: 08:00-00:30 hrs (Mon-Sun)</div>
<div><br></div>
</div>
我想在解析時排除“ Timings:08:00-00:30 hrs(Mon-Sun)” div標簽。
這是我的代碼:
import scrapy
from job.items import StarbucksItem
class StarbucksSpider(scrapy.Spider):
name = "starbucks"
allowed_domains = ["starbucks.in"]
start_urls = ["http://www.starbucks.in/coffeehouse/store-locations/"]
def parse(self, response):
for sel in response.xpath('//div[@class="region size2of3"]'):
item = StarbucksItem()
item['title'] = sel.xpath('div/text()').extract()
yield item
我將使用starts-with()
XPath函數獲取以“ Timings”開頭的div
元素的文本:
sel.xpath('.//div[starts-with(., "Timings")]/text()').extract()
請注意,頁面的HTML結構並不容易區分彼此之間的位置-沒有可重復使用的特定於位置的容器。 在這種情況下,我會找到每個h2
或strong
標簽,並使用Scrapy Shell中的 following-sibling
例子:
In [10]: for sel in response.xpath('//div[contains(@class, "region")]/*[self::h2 or self::strong]'):
name = sel.xpath('text()').extract()[0]
timings = sel.xpath('./following-sibling::div[starts-with(., "Timings")]/text()').extract()[0]
print name, timings
....:
Mumbai Timings: 08:00-00:30 hrs (Mon-Sun)
Fort Timings: 08:00-00:30 hrs (Mon-Sun)
Colaba Timings: 07:00-01:00 hrs (Mon-Sun)
Goregaon Timings: 10:00-23:30 hrs (Mon-Sun)
Powai Timings: 07:00-00:00 hrs (Mon-Sun)
...
Hi-Tech City Timings: 09:00 - 22:30 hrs (Mon - Sun)
Madhapur Timings: 11:00 -23:00 hrs (Mon - Sun)
Banjara Hills Timings: 10:00 -22:30 hrs (Mon - Sun)
另請注意,如果要提取時間范圍值,則可以使用.re()
:
In [18]: for sel in response.xpath('//div[contains(@class, "region")]/*[self::h2 or self::strong]'):
name = sel.xpath('text()').extract()[0]
timings = sel.xpath('./following-sibling::div[starts-with(., "Timings")]/text()')[0].re(r'(\d+:\d+)\s*\-\s*(\d+:\d+)')[:2]
print name, timings
Mumbai [u'08:00', u'00:30']
Fort [u'08:00', u'00:30']
Colaba [u'07:00', u'01:00']
Goregaon [u'10:00', u'23:30']
...
Hi-Tech City [u'09:00', u'22:30']
Madhapur [u'11:00', u'23:00']
Banjara Hills [u'10:00', u'22:30']
此外,請確保您在循環體內具有yield
(請參見發布的代碼)。
如果要排除Timings
並獲取其余的位置描述,請使用:
for sel in response.xpath('//div[contains(@class, "region")]/*[self::h2 or self::strong]'):
print " ".join(item.strip() for item in sel.xpath('following-sibling::div[position() < 4 and not(starts-with(., "Timings"))]/text()').extract())
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.