如何使用 Python Scrapy 从本网站获取信息？

Question

I've written this code, and I can't obtain results. This is my first time I've tried this, and I don't know what I'm doing wrong. I run and only obtain info for teams in the top of the website, not other ones.

import scrapy
from bs4 import BeautifulSoup
from scrapy.item import Field, Item
from scrapy.spiders import CrawlSpider
from scrapy.spiders import Rule
from scrapy.linkextractors import LinkExtractor
from scrapy.loader import ItemLoader
from scrapy.loader.processors import Join

class FichaClub(Item):
    nombre = Field()
    email = Field()
    zona = Field()

class SacaClubes(CrawlSpider):
    name="Spider100"
    start_urls = ["http://www.ecuafutbol.org/web/asociaciones.php"]
    allowed_domains = ['ecuafutbol.org']

    rules = (
        Rule(LinkExtractor(allow='asociacion_detalle.php*')),
        Rule(LinkExtractor(allow='club.php*'), callback= 'parse_items'),
    )

    def parse_items(self, response):
        item = scrapy.loader.ItemLoader(FichaClub(), response)
        item.add_xpath('email','//a[starts-with(@href, "mail")]/text()')
        item.add_xpath('nombre','//*[@id="gallery-post-1511"]/article/div/div/div/p/strong[1]/text()')
        yield item.load_item()

Answer 1

Correct me if I'm wrong, but it looks like you are trying to scrape the teams from the bottom table. To scrape this data you'll have to specify your parse_items to search for <div class="table-responsive"> .

You could then iterate through the list and print out / do whatever you'd like with the team names. Here is an example of what I'd try using

 soccer = BeautifulSoup(start_urls, 'html.parser')
 table = soccer.findAll("div", class_="table-responsive")
 teams = []
for line in table:
       team_found = re.findall(r'([A-Z]\w+-*\w*)', line)
       teams = teams + team_found

Try this out. If it has issues, tinker with the line table = soccer.findAll("div", class_="table-responsive") and change the class name to other elements inside of that table. Make sure to use Chromes Inspect feature to pick apart the HTML. Hope this was helpful!