Scrapy + Python + Xpath：Xpath返回一个空列表

Question

I need to scrape the links to the images from this page: http://calendar.youtoocanrun.com/events/new-delhi-1/beat-that-run/

I wrote this xpath:

response.xpath('//li[@class="geodir-active-slide"]/img/@src').extract()

It returned empty list. It should have returned the links to both gif and jpg files. Why?

Answer 1

The problem is not in your XPath expression, but in the assumption that the element you are looking for is in the page raw HTML file downloaded by Scrapy.

Scrapy doesn't run any JavaScript files so that in many cases the response you get in Scrapy is different than what you see in the developer tools.

If you open the same website using the "view page source" option from your browser, you'll see that the element you're looking for is not there. This means that such element is generated dynamically using JavaScript.

There are some ways to solve this and I'd approach it in this order:

1. check the page HTML and look for JS code containing the data you need;
2. inspect the requests that your browser is doing in the requests panel in developer tools and try to find a request that is bringing that content for you;
3. use a headless browser to render the page for you;