Python：同时多个HTTP请求

Question

I am looking to set up an HTTP server which takes in some input and then the application needs to send multiple HTTP requests at the same time (to another server). What is the best approach for this?

If I use the Twisted framework, do I still need to use threading?

Answer 1

You don't need to worry about threading or thread safety yourself. Most of the time spent doing HTTP requests is waiting for a response, at which time Python can run other code in parallel safely. Use some of the libraries and built-in functionality Python has to manage concurrent I/O waits.

You could use gevent. Twisted has similar functionality but may be more complex to learn to use. There's also the multiprocessing.dummy functionality built-in to Python.

If you go with something other than Twisted, you can and should use the requests library to do HTTP client stuff in conjunction with the above -- you may need to use grequests , which is a green-threaded variant, to do non-blocking I/O.

Answer 2

If you use the Twisted networking engine, you do not need to use threading. Just issue the requests and wait for the responses by adding callbacks that will execute as they're available.

For a nice easy way to make HTTP requests using Twisted, you might want to consider Treq , a requests -inspired convenience layer for Twisted's HTTP client.

Answer 3

Either Threading or Twisted can do this.

ie. If you use twisted, you won't need to use Threading.

Keep in mind that some servers have a limit on the number of connections they will allow from a single IP address.