Why am I getting an empty array with (scrapy shell response.xpath())?

Question

I'm wondering why response.xpath() returns an empty array [] in this page , even if I do response.xpath('//div').extract() ! Example:

$ scrapy shell https://www.amazon.cn/b/2127529051
...
>>> response.xpath('//div').extract()
[]

I can get some results from the homepage, but I can't get any results from many other pages.

BTW, I'm not trying crawling amazon or something, it's just for learning purposes only.

I tried other sites as well, but didn't face this issue, so I want to know why.

Any ideas?

thanks

Answer 1

response.xpath('\\\\div').execute()

This line is just wrong. First of all you use front-slashes instead of back-slashes here. Also execute() is not a method of Selector or SelectorList objects (those are the values that are returned by response.xpath() method).

Try: response.xpath("//div").extract()

Other than your code being broken it's a good practice to turn off javascript and run view(response) to see exactly what your spider is seeing. In some cases your spider might not even see "//div[@id='name']" because it's loaded in with javascript.

Answer 2

• response.xpath('\\div').execute()

• there can be only few reason for this

  1. User agent : scrapy shell site name -s USER_AGENT='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36'

  2. your response is empty,try just response. It shows 200-300 then its okay

  3. path is wrong as per the site

this should solve your problem