在Python中处理来自多个套接字的传入数据

Question

Background: I have a current implementation that receives data from about 120 different socket connections in python. In my current implementation, I handle each of these separate socket connections with a dedicated thread for each. Each of these threads parse the data and eventually store it within a shared locked dictionary. These sockets DO NOT have uniform data rates, some sockets get more data than others.

Question: Is this the best way to handle incoming data in python, or does python have a better way on handling multiple sockets per thread?

Answer 1

Using an asynchronous approach will make you much happier. For an example of a well-done implementation of this as a well-known application Tornado is perfect. You can easily use Tornado's ioloop for things other than web servers, too.

There are alternative libraries such as gevent ; but I believe Tornado is a better place to look at first since it both provides the loop and a web server implemented on top of it as a great example of how to use the loop well.

Answer 2

If you're using threads, that's basically the way you'd go about it.

The alternative is to use one of the various asynchronous networking libraries out there, such as Twisted , Tornado , or GEvent .

Answer 3

As mentioned in Asynchronous UDP Socket Reading question from you, asyncoro can be used to process many asynchronous sockets efficiently. Another benefit with asyncoro in your problem is that you don't need to worry about locking shared dictionary, as with asyncoro at most one coroutine is executing at any time and there is no forced preemption.