[英]does python's urllib2 do connection pooling?
Really what I'm wondering: is python's urllib2 more like java's HttpUrlConnection
, or more like apache's HttpClient
? 真的是我想知道的是:python的urllib2更像Java的HttpUrlConnection
还是更像apache的HttpClient
? And, ultimately I'm wondering if urllib2 scales when used in a http server, or if there is some alternate library that is used when performance is an issue (as is the case in the java world). 而且,最终我想知道urllib2在http服务器中使用时是否可扩展,或者在性能成为问题时是否使用某些替代库(在Java世界中就是这种情况)。
To expand on my question a bit: 为了进一步说明我的问题:
Java's HttpUrlConnection internally holds one connection open per host, and does pipelining. Java的HttpUrlConnection在内部为每个主机打开一个连接,并进行管道传输。 So if you do the following concurrently across threads it won't perform well: 因此,如果您同时在线程中执行以下操作,则效果会不佳:
HttpUrlConnection cxn = new Url('www.google.com').openConnection();
InputStream is = cxn.getInputStream();
By comparison, apache's HttpClient can be initialized with a connection pool, like this: 相比之下,可以使用连接池初始化apache的HttpClient,如下所示:
// this instance can be a singleton and shared across threads safely:
HttpClient client = new HttpClient();
MultiThreadedHttpConnectionManager cm = new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams p = new HttpConnectionManagerParams();
p.setMaxConnectionsPerHost(HostConfiguration.ANY_HOST_CONFIGURATION,20);
p.setMaxTotalConnections(100);
p.setConnectionTimeout(100);
p.setSoTimeout(250);
cm.setParams(p);
client.setHttpConnectionManager(cm);
The important part in the example above being that the number of total connections and the per-host connections are configurable. 上面示例中的重要部分是总连接数和每主机连接数是可配置的。
In a comment urllib3 was mentioned, but I can't tell from reading the docs if it allows a per-host max to be set. 在评论中提到了urllib3,但是从阅读文档中我无法确定它是否允许设置每个主机的最大值。
As of Python 2.7.14rc1, No. 从Python 2.7.14rc1开始,否。
For urllib
, urlopen()
eventually calls httplib.HTTP
, which creates a new instance of HTTPConnection
. 对于urllib
, urlopen()
最终调用httplib.HTTP
,这将创建一个 HTTPConnection
的新实例 。 HTTPConnection
is tied to a socket and has methods for opening and closing it. HTTPConnection
绑定到套接字,并具有用于打开和关闭它的方法。
For urllib2
, HTTPHandler
does something similar and creates a new instance of HTTPConnection
. 对于urllib2
, HTTPHandler
类似的操作并创建HTTPConnection
的新实例。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.