从脚本运行scrapy时出错

Question

I'm trying to run a scrapy spider from script instead of running it from the command terminal like this:

scrapy crawl spidername

In the scrapy documentation I found the following example: https://doc.scrapy.org/en/latest/topics/practices.html .

Now, my code looks like this:

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.loader import ItemLoader
from properties.items import PropertiesItem


class MySpider(scrapy.Spider):
    name = "basic"
    allowed_domains = ["web"]
    start_urls = ['http://www.example.com']

    def parse(self, response):
        l = ItemLoader(item=PropertiesItem(), response = response)
        l.add_xpath('title', '//h1[1]/text()')

        return l.load_item()

process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished

When I run this script I get the following error:

File "/Library/Python/2.7/site-packages/Twisted-16.7.0rc1-py2.7-macosx-10.11-intel.egg/twisted/internet/_sslverify.py", line 38, in TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1, AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'

So my questions are:

1) What kind of error is this? I haven't been able to find any examples online.

2) What can I change to make scrapy run from this script?

Updated:

Added packages installed for project

attrs==16.3.0 
Automat==0.3.0 
cffi==1.9.1 
characteristic==14.3.0 
constantly==15.1.0 
cryptography==1.7.1 
cssselect==1.0.0 
enum34==1.1.6 
idna==2.2 
incremental==16.10.1 
ipaddress==1.0.17 
lxml==3.7.1 
parsel==1.1.0 
pyasn1==0.1.9 pyasn1-
modules==0.0.8 
pycparser==2.17 
PyDispatcher==2.0.5 
pyOpenSSL==0.15.1 
queuelib==1.4.2 
Scrapy==1.3.0 service-
identity==16.0.0 
six==1.10.0 
tree==0.1.0 
Twisted==16.6.0 
virtualenv==15.1.0 
w3lib==1.16.0 zope.
interface==4.3.3

Answer 1

I found a solution:

Created a new virtual environment that's based on python 3.6 instead of python 2.7. I ran the exact same code (had to replace urlparse with urllib.parse though) and it worked!

Answer 2

1) I am not sure

2) But your indentation is needs a review:

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.loader import ItemLoader
from properties.items import PropertiesItem


class MySpider(scrapy.Spider):
    name = "basic"
    allowed_domains = ["web"]
    start_urls = ['http://www.example.com']

    def parse(self, response):
        l = ItemLoader(item=PropertiesItem(), response = response)
        l.add_xpath('title', '//h1[1]/text()')

        return l.load_item()

    process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

    process.crawl(MySpider)
    process.start() # the script will block here until the crawling is finished

and I am assuming various other things a examples in the code. ie to run the below spider you will need to input

scrapy crawl basic

and that you have a folder called "properties" with the file "items" in it etc