Python Web Scraping-導航到下一頁鏈接並獲取數據

Question

我正在嘗試導航到鏈接並提取數據（該數據是href下載鏈接），該數據應添加到除第一頁上的先前字段（從中獲得鏈接的位置）之外的新字段中，但我正在努力怎么做

首先，我已經創建了一個解析器，並提取了第一頁的所有鏈接，並將其添加到名為“鏈接”的字段中，此鏈接將重定向到包含下載按鈕的頁面，因此我需要下載的真實鏈接按鈕，所以我在這里所做的是用先前的鏈接創建一個for循環並執行函數yield response.follow，但操作並不順利。

import scrapy
class thirdallo(scrapy.Spider):
    name = "thirdallo"
    start_urls = [
        'https://www.alloschool.com/course/alriadhiat-alaol-ibtdaii',

]

def parse(self, response):

        yield {
          'path': response.css('ol.breadcrumb li a::text').extract(),
          'links': response.css('#top .default .er').xpath('@href').extract() 

        }

        hrefs=response.css('#top .default .er').xpath('@href').extract()
        for i in hrefs:
            yield response.follow(i, callback=self.parse,meta={'finalLink' :response.css('a.btn.btn-primary').xpath('@href)').extract() })

Answer 1

在@href您試圖抓取，似乎您有一些.rar鏈接，這些鏈接無法使用指定的函數進行解析。

在下面找到我的代碼，包括requests和lxml庫：

>>> import requests
>>> from lxml import html
>>> s = requests.Session()
>>> resp = s.get('https://www.alloschool.com/course/alriadhiat-alaol-ibtdaii')
>>> doc = html.fromstring(resp.text)
>>> doc.xpath("//*[@id='top']//*//*[@class='default']//*//*[@class='er']/@href")
['https://www.alloschool.com/assets/documents/course-342/jthathat-alftra-1-aldora-1.rar', 'https://www.alloschool.com/assets/documents/course-342/jthathat-alftra-2-aldora-1.rar', 'https://www.alloschool.com/assets/documents/course-342/jthathat-alftra-3-aldora-2.rar', 'https://www.alloschool.com/assets/documents/course-342/jdadat-alftra-4-aldora-2.rar', 'https://www.alloschool.com/element/44905', 'https://www.alloschool.com/element/43081', 'https://www.alloschool.com/element/43082', 'https://www.alloschool.com/element/43083', 'https://www.alloschool.com/element/43084', 'https://www.alloschool.com/element/43085', 'https://www.alloschool.com/element/43086', 'https://www.alloschool.com/element/43087', 'https://www.alloschool.com/element/43088', 'https://www.alloschool.com/element/43080', 'https://www.alloschool.com/element/43089', 'https://www.alloschool.com/element/43090', 'https://www.alloschool.com/element/43091', 'https://www.alloschool.com/element/43092', 'https://www.alloschool.com/element/43093', 'https://www.alloschool.com/element/43094', 'https://www.alloschool.com/element/43095', 'https://www.alloschool.com/element/43096', 'https://www.alloschool.com/element/43097', 'https://www.alloschool.com/element/43098', 'https://www.alloschool.com/element/43099', 'https://www.alloschool.com/element/43100', 'https://www.alloschool.com/element/43101', 'https://www.alloschool.com/element/43102', 'https://www.alloschool.com/element/43103', 'https://www.alloschool.com/element/43104', 'https://www.alloschool.com/element/43105', 'https://www.alloschool.com/element/43106', 'https://www.alloschool.com/element/43107', 'https://www.alloschool.com/element/43108', 'https://www.alloschool.com/element/43109', 'https://www.alloschool.com/element/43110', 'https://www.alloschool.com/element/43111', 'https://www.alloschool.com/element/43112', 'https://www.alloschool.com/element/43113']

在您的代碼中，嘗試以下操作：

for i in hrefs:
    if '.rar' not in i:
        yield response.follow(i, callback=self.parse,meta={'finalLink' :response.css('a.btn.btn-primary').xpath('@href)').extract() })