[英]How do I use the Python Scrapy module to list all the URLs from my website?
I want to use the Python Scrapy module to scrape all the URLs from my website and write the list to a file.我想使用 Python Scrapy 模块从我的网站上抓取所有 URL 并将列表写入文件。 I looked in the examples but didn't see any simple example to do this.
我查看了示例,但没有看到任何简单的示例来执行此操作。
Here's the python program that worked for me:这是对我有用的python程序:
from scrapy.selector import HtmlXPathSelector
from scrapy.spider import BaseSpider
from scrapy.http import Request
DOMAIN = 'example.com'
URL = 'http://%s' % DOMAIN
class MySpider(BaseSpider):
name = DOMAIN
allowed_domains = [DOMAIN]
start_urls = [
URL
]
def parse(self, response):
hxs = HtmlXPathSelector(response)
for url in hxs.select('//a/@href').extract():
if not ( url.startswith('http://') or url.startswith('https://') ):
url= URL + url
print url
yield Request(url, callback=self.parse)
Save this in a file called spider.py
.将其保存在一个名为
spider.py
的文件中。
You can then use a shell pipeline to post process this text:然后,您可以使用 shell 管道对该文本进行后期处理:
bash$ scrapy runspider spider.py > urls.out
bash$ cat urls.out| grep 'example.com' |sort |uniq |grep -v '#' |grep -v 'mailto' > example.urls
This gives me a list of all the unique urls in my site.这为我提供了我网站中所有唯一网址的列表。
something cleaner (and maybe more useful) would be using LinkExtractor更干净(也许更有用)的东西会使用LinkExtractor
from scrapy.linkextractors import LinkExtractor
def parse(self, response):
le = LinkExtractor() # empty for getting everything, check different options on documentation
for link in le.extract_links(response):
yield Request(link.url, callback=self.parse)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.