Scrapy 中的 Xpath 或 css 选择器属性值

Question

Hi there I am new to scrapy and I want to extract an attribute value form an html element. So what could be the right way to extract that attribute value form that html . I want to extract "data-next-url" attribute

<div class="loading_more_jobs" data-type="loading_more_jobs" style="display:none;" data-next-url="https://www.ziprecruiter.com/candidate/search?search=restaurant&amp;page=2&amp;location=Atlanta%2C+Georgia"></div>

I am using that xpath but it is not working

 response.xpath('//*[@class="loading_more_jobs"]/@data-next-url').extract()

Answer 1

If you check source HTML you'll find this:

  <button class="load_more_jobs" data-type="load_more_jobs" data-next-url="">Load More Job Results</button>
  <div class="loading_more_jobs" data-type="loading_more_jobs" style="display:none;"></div>

But you can get next page URL anyway:

<div class="job_results" data-this-url="/candidate/search?search=restaurant&amp;location=Atlanta%2C+Georgia" data-next-url="/candidate/search?location=Atlanta%2C+Georgia&amp;page=2&amp;search=restaurant" data-type="job_results">

=>

response.xpath('//div[@class="job_results"]/@data-next-url').extract_first()

or

<link rel="next" href="https://www.ziprecruiter.com/candidate/search?location=Atlanta%2C+Georgia&amp;page=2&amp;search=restaurant">

=>

response.xpath('//link[@rel="next"]/@href').extract_first()