[英]getting while trying to parse page with scrapy
當我嘗試獲取所有頁面內容時,我在控制台中收到此錯誤
2018-11-08 20:55:34 [scrapy.core.engine] INFO: Spider opened
2018-11-08 20:55:34 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-11-08 20:55:34 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-11-08 20:55:34 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
File "c:\python36\lib\site-packages\scrapy\core\engine.py", line 127, in _next_request
request = next(slot.start_requests)
File "c:\python36\lib\site-packages\scrapy\spiders\__init__.py", line 83, in start_requests
yield Request(url, dont_filter=True)
File "c:\python36\lib\site-packages\scrapy\http\request\__init__.py", line 25, in __init__
self._set_url(url)
File "c:\python36\lib\site-packages\scrapy\http\request\__init__.py", line 62, in _set_url
raise ValueError('Missing scheme in request url: %s' % self._url)
這是我的代碼的樣子
import scrapy
class Shopee(scrapy.Spider):
name = 'Shopee'
start_urls = ['http://www.shopee.sg/Games-Hobbies-cat.14']
def parse(self, response):
print(response.text)
您帖子中的錯誤消息與start_urls
缺少http(s)://
。 我想您在更新代碼時忘記更新錯誤消息。
但是在運行您的代碼后,該站點似乎正在阻止基於User-Agent
客戶端。 考慮嘗試瀏覽器的用戶代理字符串。 例如:
name = 'Shopee'
start_urls = ['http://www.shopee.sg/Games-Hobbies-cat.14']
custom_settings = {
'DEFAULT_REQUEST_HEADERS': {
'User-Agent': (
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14'
' (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A'
)
}
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.