How can I pass my ID and my password to a website in Python using Google App Engine?

Question

Here is a piece of code that I use to fetch a web page HTML source (code) by its URL using Google App Engine:

from google.appengine.api import urlfetch
url = "http://www.google.com/"
result = urlfetch.fetch(url)
if result.status_code == 200:
   print "content-type: text/plain"
   print
   print result.content

Everything is fine here, but sometimes I need to get an HTML source of a page from a site where I am registered and can only get an access to that page if I firstly pass my ID and password. (It can be any site, actually, like any mail-account-providing site like Yahoo: https://login.yahoo.com/config/mail?.src=ym&.intl=us or any other site where users get free accounts by firstly getting registered there). Can I somehow do it in Python (trough "Google App Engine")?

Answer 1

您可以检查HTTP状态代码401，“需要授权”，并提供网站要求的HTTP授权类型（基本，摘要等） - 请参阅此处了解更多详细信息（没有太多的GAE具体在这里 - 这是一个学习HTTP细节和遵守它们的问题！ - ）。

Answer 2

正如亚历克斯所说，您可以检查状态代码并查看它想要的自动化类型，但您无法概括它，因为某些网站不会给出任何提示或仅允许通过非标准表单登录，在这些情况下您可能需要自动化使用表单的登录过程，您可以使用twill（ http://twill.idyll.org/ ）等库或为每个站点编写特定表单提交代码。