Scrapy：如何抓取从下拉列表中选择的多个网页

Question

How would I go about scraping through multiple pages of one website that are navigated through a dropdown and button with scrapy? I know how to do pagination by finding the link to the next page in the current page, but the same technique cannot be applied for this situation.

One idea I've thought of is finding the value of the next dropdown option and changing the url to direct to the respective page. Would this be a valid solution?

Here's an example: http://highschoolsports.nj.com/football/standings/?grouping=15764

Answer 1

Firstly visit one of the pages about teams. Such as GMC Blue then get all the value s from dropdown selects.

 <option value="">Select a Conference - Division</option>
 <option value="15764" selected="selected">GMC - Blue</option>
 <option value="15767">GMC - Red</option>
 <option value="15713">GMC - White</option>
 <option value="18380">Independent</option>
 <option value="15773">Mid-State 38 - Delaware</option>
 <option value="15854">Mid-State 38 - Mountain</option>
 <option value="15824">Mid-State 38 - Raritan</option>
  ....

http://highschoolsports.nj.com/football/standings/?grouping=18380 create a for loop and change value of grouping in each request.