POST请求后使用Python解析网站

Question

I want to scrap some addresses off of a store locator for a website to make it easier to put into google maps and see where all the stores in my state are located.

Here is the store locator website: http://www.rockyboots.com/locator

When I go here, I put in my state, and it returns all the addresses. I am wondering how to do this with python, specifically the Requests module.

I tried to reverse engineer what information was being sent when I submitted the form, and got

form = {'dwfrm_storelocator_address_states_stateUSCA':'NS', 
        'dwfrm_storelocator_findbystate':'Search'}

so my request currently looks like:

r = requests.post(url, data = form)

where

url = 'http://www.rockyboots.com/locator'

This seems to just give me the locator page again, not the submitted form page.

At this point, I am just not sure what else to do or try, so any info would be helpful. This is just a side project that I could use to learn some web scraping.

Answer 1

A user on Reddit ( commandlineluser ) helped me with the solution. Turns out, there was another parameter that had to be passed to the POST request, but that parameter changed every time you access the URL.

The final solution is shown here.