简体繁体 English

有没有一种简便的方法来生成JavaScript？

[英]Is there an easy and fast way to generate JavaScript?

原文 2015-05-04 13:27:51 7 2 python/ ajax/ web-crawler/ scrapy

My problem begins when i try to crawl an app store, lets say google play. 当我尝试爬网应用商店时，我的问题开始了，比如说Google Play。 for every app there are alot of comments and i want to crawl them FAST. 对于每个应用程序，都有很多评论，我想快速检索它们。 but the comment section in google is generated by java script. 但是Google中的注释部分是由Java脚本生成的。 here is a link for example: https://play.google.com/store/apps/details?id=com.gameloft.android.ANMP.GloftAMHM in that link you can see that in order to generate more comments you need to click on a button several times. 例如，这是一个链接： https : //play.google.com/store/apps/details?id=com.gameloft.android.ANMP.GloftAMHM在该链接中，您可以看到，为了生成更多评论，您需要单击一个按钮几次。 (after 5-6 clicks aprox) the page generate more comments by executing a javascript. （5-6次点击aprox之后），该页面会通过执行JavaScript生成更多评论。

At first i solved this problem using a web driver (firefox) and simulate a real person clicking on the button, and it generate comments, and he keep pressing till all comments are generated. 最初，我使用Web驱动程序（firefox）解决了这个问题，并模拟了一个真实的人单击该按钮，然后生成注释，然后他一直按下直到生成所有注释。

Problem with this is: 1, it takes too much time. 问题是：1，花费太多时间。 2, sometimes after tons fo clicks and JS generation the web browser is fail to response. 2，有时在点击和生成JS之后，Web浏览器无法响应。

What I need is a way to generate all comments per application in a better, faster way. 我需要的是一种以更好，更快的方式为每个应用程序生成所有注释的方法。 maybe theres some kind of tech, or just anything else that would improve my solution, 也许有某种技术，或者其他任何可以改善我的解决方案的技术，

Im using a spider I've created in scrapy. 我正在使用我在草皮中创建的蜘蛛。

All kind of help will be much appreciated 各种各样的帮助将不胜感激

2 个解决方案

One of the reasons they generate/show additional comments is exactly that they do not want someone to crawl them... the other is for the initial page to load without them (faster), and only if someone starts reading comments to show few more.. 他们生成/显示其他评论的原因之一就是他们不想让某人抓取它们…另一个原因是在没有他们的情况下（更快）加载初始页面，并且只有当有人开始阅读评论以显示更多评论时..

Unless they provide an API where you can pull all the comments at once, I do not see another quick way of pulling them, apart of simulating clicks and scrolls... (slow way of doing it) 除非它们提供一个可以一次提取所有注释的API，否则除了模拟点击和滚动之外，我看不到另一种快速提取注释的方法……（这样做很慢）

Are you respecting robots.txt ? 您尊重robots.txt吗？ Why or why not? 为什么或者为什么不？