简体繁体 English

如何使用Python（Gevent）异步获取内容？

[英]How to grab contents asynchronously using Python(Gevent)?

原文 2012-09-14 04:59:42 7 1 python/ celery/ gevent

The scenario is save the response of an API request using RMDB id as a parameter. 该方案将使用RMDB id作为参数保存API请求的响应。 I want to grab all the movie info from imdv-id tt0000001 to tt9999999. 我想获取从imdv-id tt0000001到tt9999999的所有电影信息。

Now I'm using gevent to run several threads(gevent.joinall(threads)), it's not so fast. 现在，我正在使用gevent运行多个线程（gevent.joinall（threads）），但速度不是很快。 Is there other solutions for this kind of problems, like using Celery+RabbitMQ? 还有其他解决此类问题的解决方案，例如使用Celery + RabbitMQ？

1 个解决方案

For one you must make sure that you aren't making any blocking calls in your code, as that will also block everything else from running, slowing the entire system. 首先，您必须确保您没有在代码中进行任何阻塞调用，因为这也会阻塞其他所有程序的运行，从而降低整个系统的速度。 Reasons for blocking include tight loops or IO that has not been patched by eventlet's monkey patch (eg C extensions). 阻塞的原因包括紧密循环或未由eventlet的猴子补丁（例如C扩展）补丁的IO。

Celery supports using eventlet & gevent, and that is probably the recommended concurrency option for what you are doing (web request IO). Celery支持使用eventlet和gevent，这可能是您正在执行的操作（Web请求IO）的推荐并发选项。 Celery may not make your code run faster though, but it enables you to easily distribute the work to many machines. 尽管Celery可能不会使您的代码运行得更快，但是它使您能够轻松地将工作分配给许多计算机。

To optimize you should always profile your code to find out what the bottleneck is. 为了进行优化，您应该始终对代码进行概要分析，以找出瓶颈所在。 It could be many things, eg slow network, slow host, slow DNS or something else entirely. 可能有很多事情，例如缓慢的网络，缓慢的主机，缓慢的DNS或完全其他的东西。