Scrapy Start_URL不正确

Question

So i'm new to scrapy and I'm coming to an issue where (I believe) the start URL isn't correct.

Then click the links to load into the camps description

However when I use that start URL it doesnt load. Meaning That scrapy opens and loads the telnet but will never connect. When I use http://www.w3.org/1999/xhtml (which I get from the top line in inspect (chrome) It crawls but seems that is the completely wrong site. ( I got this link from the top of the inspect page)

and where it SHOULD start, the URL is: http://www.kidscamps.com/camps/california-overnight-camps-page0.html

Any ideas? and Thanks in advance! Sorry about all the commented out

So I guess my biggest question is how do I find the CORRECT url to start with since all my other scripts work correctly.

Also it doesn't work without rules assigned.

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from scrapy.selector import Selector
from kidscamp_com.items import KidscampComItem
import html2text


class MySpider(CrawlSpider):
    name = "kids"
    #allowed_domains = "http://www.bayareaparent.com/Camp-Guide/index.php/cp/1/si/0/"
    start_urls = ['http://www.kidscamps.com/residential/overnight_camp.html'
    ]

    rules = (
    Rule(LinkExtractor(allow=(), restrict_xpaths=('//*[@id="results-wrapper"]/div[1]/p[1]/a',)), callback="parse1", follow=True),
     )


    def parse1(self, response):
        hxs = Selector(response)
        body = hxs.xpath('//*[@id="body-wrapper"]')
        items = []
        for body in body:
            item = KidscampComItem()
         #   item["camp_name"] = body.xpath('').extract()
          #  item["location"] = body.xpath('').extract()
            item["phone"] = body.xpath('//a[@class="phone"]//text()').extract()
            item["website"] = body.xpath('//*[@id="results-wrapper"]/div[1]/div/div[2]/ul[2]/li[2]/a').extract()
           # item["email"] = body.xpath('').extract()
            item["description"] = body.xpath('//*[@id="info-page"]/div[2]/div//text()').extract()
            item["camp_size"] = body.xpath('//*[@id="info-page"]/div[2]/div/ul[1]/li[1]/dd').extract()
            item["founded"] = body.xpath('//*[@id="info-page"]/div[2]/div/ul[1]/li[2]/dd').extract()
            item["gender"] = body.xpath('//*[@id="info-page"]/div[2]/div/ul[1]/li[3]/dd').extract()
            item["maximum_age"] = body.xpath('//*[@id="info-page"]/div[2]/div/ul[2]/li[1]/dd').extract()
            item["minimum_age"] = body.xpath('//*[@id="info-page"]/div[2]/div/ul[2]/li[2]/dd').extract()
            item["nearest_city"] = body.xpath('//*[@id="info-page"]/div[2]/div/ul[2]/li[3]/dd').extract()
            items.append(item)
            return items

Checked out robots.txt which should allow crawling over most of their site. However after reading into the source a little more I noticed this line:

does that mean that even though its not on /robots.txt its still not considered allowed? I even tried without listening to robots (to see if anything changed) and Nothing different happened. But if someone does know that answer that would be cool.

UPDATE

Found out that when i changed: start_urls to start_url it works. The thing thats weird about this is that I have used start_urls for my other spiders and it works regardless of the (s). Wonder why its changing anything here

Answer 1

Bo scrapy standard spider class scrapy.spiders.Spider and the class scrapy.spiders.CrawlSpider use the attribute start_urls .

From the official documentation :

class MySpider(CrawlSpider):
    name = 'example.com'
    allowed_domains = ['example.com']
    start_urls = ['http://www.example.com']
    ...

class MySpider(scrapy.Spider):
    name = 'example.com'
    allowed_domains = ['example.com']
    start_urls = ['http://www.example.com/1.html']
    ...

The attribute start_url is not used anywhere.

Answer 2

Seems like website you're crawling doesn't work well with Scrapy default user agent.

Make sure site is ok with you crawling them, if they are ok agree some UA with them so that they know it's you. Setting user agent in scrapy is matter of setting user_agent spider attribute, for example:

class MySpider(Spider):
    user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36"