Scrapy shell response.xpath returning blank array [ ]

Question

The xpath is:

/html/body/form/div[12]/div/div[5]/div/div[3]/div[1]/div[13]/ul/li[*]/div/ul/li/ul/li[2]/div[3]

I am getting the output through Chrome Console through the string below:

$x('/html/body/form/div[12]/div/div[5]/div/div[3]/div[1]/div[13]/ul/li[*]/div/ul/li/ul/li[2]/div[3]/text()')

But when I type response.xpath instead of $s and include .extract() at the end in Scrapy shell, it returns a blank array [].

>>> response.xpath('/html/body/form/div[12]/div/div[5]/div/div[3]/div[1]/div[13]/ul/li[*]/div/ul/li/ul/li[2]/div[3]/text()').extract()

[]

Can someone please help. Thanks in advance

PS: I know it's an absolute path. Want to understand why it's working in Chrome Console, and not in Scrapy Shell

Answer 1

some times body tag doesn't exist in source, but browser add it. so xpath doesn't work using body tag

so maybe you can give a try to

/html//form/div[12]/div/div[5]/div/div[3]/div[1]/div[13]/ul/li[*]/div/ul/li/ul/li[2]/div[3]/text()

on the other hand your xpath looks like very hard coded, it should be flexible enough .. you should read out a xpath tutorial and make it simple using Elements @id and other attributes