Unable to scrape from dropdown list using Scrapy

Question

I am trying to scrape a list of markets from a JS dropdown list embedded on a website: https://e27.co/startups

Using scrapy shell, I tried to scrape the list of markets from the 'Markets' dropdown menu but unable to do so.

After running scrapy shell 'https://e27.co/startups' , I tried using both response.css() as well as response.xpath() .

For css selector:

response.css('#startups-page > div > div.search-block.box-view > div.row.mbt-s > div > div > ul > li:nth-child(3)')

For xpath, I tried:

response.xpath('//*[@id="startups-page"]/div/div[1]/div[2]/div/div/ul/li[3]/a"')

Both are obtained from inspecting the dropdown element.

However, an empty list is returned.

May I know how to scrape all the different markets from the dropdown list? Thanks.

Answer 1

This data is located in separate small request to https://e27.co/startups?json .

From scrapy shell "https://e27.co/startups?json" I could get whole list with this expression:

In [1]: response.css('select#market option::text').extract()
Out[1]: 
[u'Advertising',
 u'Aerospace',
 u'Agency & Consulting',
 u'Agritech',
 u'Architecture & Construction',
...