retrieving a page that redirects to a login page within python

Question

I am having a rough time gathering the data from a website programatically. I am attempting to utilize this example to log into the server, but it is not working since I think that this is the wrong type of login.

The site I am trying to access redirects to a login page when I attempt to download the data to parse the html.

This is the URL:

https://mtred.com/rewards.html

and heres the code:

# build opener with HTTPCookieProcessor
o = urllib2.build_opener( urllib2.HTTPCookieProcessor() )
urllib2.install_opener( o )
# assuming the site expects 'user' and 'pass' as query params
p = urllib.urlencode( { 'UserLogin_username': 'mylogin', 'UserLogin_password': 'mypass' } )
# perform login with params
f = o.open( 'http://www.mtred.com/user/login.html',  p )
data = f.read()
f.close()
# second request should automatically pass back any
# cookies received during login... thanks to the HTTPCookieProcessor
f = o.open( 'https://www.mtred.com/rewards.html',p )
data = f.read()
print data

it kicks me to the login page again when I attempt to open rewards. I am trying to pass the rewards to do some statistics automatically since this information isn't available via public API

Answer 1

One issue that pops out is that you're passing in the id values of the form parameters for the login, not the name parameters. Eg, in the user name form field, you are specifying UserLogin_username , but the name of that field as expected by the server is "UserLogin[username]"

<label for="UserLogin_username" class="required">
username or email <span class="required">*</span></label>       
<input name="UserLogin[username]" id="UserLogin_username" type="text" />    </div>

<div class="row">
<label for="UserLogin_password" class="required">password <span class="required">*</span></label>   
<input name="UserLogin[password]" id="UserLogin_password" type="password" /> </div>

Since the server isn't getting back parameters that it knows about, the behavior you're seeing is not unexpected. (Not saying that there aren't other problems here; haven't looked.)

Answer 2

you must inclue in ur post data the value named "YII_CSRF_TOKEN" that included in html form . or use " ClientForm " lib