Using Scrapy to scrape a webpage

Question

I am trying to scrape the largest table from this page. I am working in conda prompt

http://www.tennisabstract.com/cgi-bin/player-classic.cgi?p=RafaelNadal&f=ACareerqq

Here is my attempt:

scrapy shell
fetch('http://www.tennisabstract.com/cgi-bin/player-classic.cgi?p=RafaelNadal&f=ACareerqq')
table = response.xpath('//table[@id="matches"]')
rows = table.xpath('//tr')

This seems to grab the correct table, but the rows I'm getting are not correct; they are coming from the top of the webpage.

My other try was this:

table = response.xpath('//table[@id="matches"]//tr')

But this just returns an empty list.

Any help is appreciated

Thanks

Answer 1

table.extract() is returning empty table ['<table id="matches"></table>'] here. It seem data getting loaded dynamically in the table. You would require to use Selenium or ScrapyJS to get data in such cases. See the link for more info