Twisted getpage using proxy

Question

I use getpage () to load the pages:

 d = getPage(url)
 d.addCallback(parsePage,url)
 d.addErrback(downloadError,url) 

Now you need to download via http proxy. How can I call getpage () to use http proxy?

Answer 1

Use twisted.web.client.ProxyAgent instead. getPage is Twisted's old, not-very-good HTTP client API. IAgent is the new, better HTTP client API. Apart from its other advantages, it also has more features than getPage - including support for HTTP proxies.

Here's an example:

from __future__ import print_function

from os import environ

from twisted.internet.task import react
from twisted.internet.endpoints import HostnameEndpoint
from twisted.web.client import ProxyAgent

def main(reactor, proxy_hostname):
    endpoint = HostnameEndpoint(reactor, proxy_hostname, 80)
    agent = ProxyAgent(endpoint)
    return agent.request(b"GET", b"http://google.com/").addCallback(print)

react(main, [environ["HTTP_PROXY"]])