How do you get the scraped data using Scrapy with javascript changes?

Question

Javascript is hiding some elements but when I am scraping the data using scrapy those elements are scraped as if they are not hidden by Javascript.

---

Link 1: http://www.limeroad.com/turquoise-sling-ivory-tag-p38444 Link 2: http://www.limeroad.com/baby-pink-wallet-eske-p35768#pOverlay=none

---

For Example: When I crawled these two links using Scrapy with xpath (//div[@id='productDetails']/ul[@class='details']/li[@id='add_to_cart_li']/a[@id='add_to_cart']/span[@class='buy-txt']), I got "Buy now" as a text in both the links even though one link doesnot contain "buy now" Text. This is happening because Javascript is changing the display . Scrapy doesn't load the Javascript changed content.

Answer 1

Not sure how the elements are hidden. see. What is the difference between visibility:hidden and display:none?

If you are using xpath to extract the data then something like this( http://www.w3schools.com/css/css_display_visibility.asp ) would help. Mind you passing the link to give an idea how to use xpath for visible elements.

Hope this is of any help.