使用python twisted写一个web爬虫

Question

I'm using Twisted to write a web crawler driven with Selenium . The idea is that I spawn twisted threads for a twisted client and a twisted server that will proxy HTTP requests to the server. Something that looks like this:

    +--------+       +--------+
    |        +------>+        |
 -->| Client |       | Server |---> WWW
    |        +<------+        |
    +--------+       +--------+

All this is running in the same process, though. The question is whether twisted allows this kind of applications or is only thought to run Client and Server as different processes (as this is the typical case I've seen everywhere).

Answer 1

You can't use Twisted in a thread. You can, however, make a single Twisted thread which can happily make multiple clients and servers.

You may need to describe your problem in a bit more detail for a better answer than that.