[英]Scrapy, scrape from drop down menu
I am trying to scrape the names of some towns/cities from a drop down menu. 我正在尝试从下拉菜单中刮取某些城镇的名称。
The html looks like: html看起来像:
option value="/salaries/ THIS_IS_WHAT_I_WANT " data-tn-link data-tn-element="loc_city[]"> Acton 选项value =“ / salaries / THIS_IS_WHAT_I_WANT ” data-tn-link data-tn-element =“ loc_city []”> Acton
I have tried this: 我已经试过了:
def parse_cities(self, response):
for city in response.xpath('//select[@id="cmp-salary-loc-select"]/option/text()').extract():
print(scrapy.Request(response.urljoin("/"+city)))
With no luck. 没有运气。
I know that this 我知道
response.xpath('//select[@id="cmp-salary-loc-select"]/option').extract()
Returns all the html I need, But just can't seem to get the values? 返回我需要的所有html,但是似乎无法获取值?
Any pointers would be appreciated. 任何指针将不胜感激。 I haven't posted the link as I do want to try figure this one out rather than someone do it for me! 我还没有发布链接,因为我想尝试找出这个链接,而不是有人替我做! But will add in if you guys think it would help? 但是,如果你们认为这会有所帮助,会补充吗?
def parse_cities(self, response):
for city in response.xpath('//select[@id="cmp-salary-loc-select"]/option/@value').extract():
yield scrapy.Request(response.urljoin("/"+city), callback=some_method)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.