Scrapy：爬虫不爬行

Question

I'm obviously new to python, scrapy and programming in general. I'm trying to scrape this site but my code doesn't seem to work. All the examples and tutorials I found deal with simple and plain websites. Or maybe I just can't get my head around it. There are hundreds of results I need to scrape, and I really don't want to do it manually.

So at this instance im just trying to only get the href from the div object to check if it works. It doesn't.

import scrapy
import requests


class QuotesSpider(scrapy.Spider):
    name = "items"

    def start_requests(self):
        urls = [
            'https://www.bosch-professional.com/ar/es/dl/localizador-de-distribuidores/dealerslist/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        list_doc = open('list_doc.txt', 'w')
        for item in response.css('div.row.m-dealer_list__row'):
            yield {
                'text': item.css('a::attr(href)').extract(),

            }

When run on the terminal (ignoring robots) it returns:

2019-01-30 23:57:13 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
2019-01-30 23:57:13 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-01-30 23:57:13 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-01-30 23:57:13 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-01-30 23:57:13 [scrapy.core.engine] INFO: Spider opened
2019-01-30 23:57:13 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-01-30 23:57:13 [scrapy.extensions.telnet] DEBUG: Telnet console listening on #NUMBER
2019-01-30 23:57:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.bosch-professional.com/ar/es/dl/localizador-de-distribuidores/dealerslist/> (referer: None)
2019-01-30 23:57:16 [scrapy.core.engine] INFO: Closing spider (finished)
2019-01-30 23:57:16 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 276,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 70592,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 1, 31, 2, 57, 16, 541215),
 'log_count/DEBUG': 2,
 'log_count/INFO': 7,
 'memusage/max': 57974784,
 'memusage/startup': 57974784,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2019, 1, 31, 2, 57, 13, 861593)}
2019-01-30 23:57:16 [scrapy.core.engine] INFO: Spider closed (finished)

Thanks for any help you can provide.

Answer 1

As far as I can see, there are really no such elements on the page:

In [2]: fetch("https://www.bosch-professional.com/ar/es/dl/localizador-de-distribuidores/dealerslist/")
2019-01-31 09:31:47 [scrapy.core.engine] INFO: Spider opened
2019-01-31 09:31:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.bosch-professional.com/ar/es/dl/localizador-de-distribuidores/dealerslist/> (referer: None, latency: 0.87 s)

In [3]: response.css('div.row.m-dealer_list__row')
Out[3]: []

But if you will try:

In [4]: response.css('div.m-dealer_citylist__card a::text').extract()
Out[4]: 
[u'25 DE MAYO - BS AS',
 u'25 DE MAYO - LA PAMP',
 u'25 DE MAYO',
 u'9 DE ABRIL',
...
 u'ZENON PEREYRA',
 u'Z\xc1RATE']

Answer 2

I have visited the website you are trying to scrap, but your css does not seems to be matching any attribute in HTML.

There is no any tag with the class name of m-dealer_list__row

All i see is m-dealer_citylist

If i describe your css:

You css describes that you are extracting a div element with 2 classes one is row and the second one is m-dealer_list__row

if you want a div with row class then in this div any tag with m-dealer_list__row you can try this:

"div.row .m-dealer_list__row"

Answer 3

It is for you). understand it yourself

import scrapy
import requests


class QuotesSpider(scrapy.Spider):
    name = "items"

    def start_requests(self):
        urls = [
            'https://www.bosch-professional.com/ar/es/dl/localizador-de-distribuidores/dealerslist/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        list_doc = open('list_doc.txt', 'w')
        for item in response.css('.trackingTeaser'):
            href = item.css('a::attr(href)').extract_first()
            href = response.urljoin(href)
            list_doc.write(''.join(item.css('a::text').re(r'([^ \n\t]+)')) + ': ' + href + '\n')
        list_doc.close()

OUTPUT: