Python Scrapy: web-scraping asp site

Question

I have scrapped many other site with needed form input, but i'm struggling to figure this one out..

http://search.appleone.com/ResumeSearch/results.asp

When I search for something, the website returns an arbitrary results.asp file that's not specific to the search term. What I am trying to do is, input a search, scrape the results page. I'm struggling with the input a search part. Typically, I would do something like http://bdomainnameh.com/search/ input search

I'd appreciate any help

Answer 1

The form has a hidden input (line 285:

<form name="frmAE" action="process.asp?page=SearchDetailed" method="POST">
<input type=hidden name="hdnAction" value="">

When the "Next>>" button is clicked, it calls sendForm(2, 0) (line 428) which assigns a value of 2 to the hidden input (line 245) before submitting it as a POST request (not GET, which is why the results page is "not specific to the search term" ie does not show the search terms in the url).

You need to (a) add this hidden value to your request and (b) submit a POST request, not GET.