Scrapy - 如何使用 AWS Lambda 函数运行爬虫？

Question

Currently I have two small projects using Scrapy. One project is basically to scrape URL's, while the other is only to scrape products of the scrape URL's. The directory structure is this:

.
├── requirements.txt
├── .venv
├── url
|   ├── geckodriver
|   ├── scrapy.cfg
|   ├── url
|   |   ├── items.py
|   |   ├── middlewares.py
|   |   ├── pipelines.py
|   |   ├── settings.py
|   |   ├── spiders
|   |   |    ├── store1.py
|   |   |    ├── store2.py
|   |   |    ├── ...
├── product
|   ├── geckodriver
|   ├── scrapy.cfg
|   ├── product
|   |   ├── items.py
|   |   ├── middlewares.py
|   |   ├── ...

When I want to run a spider using the command, I always must follow this path: ~/search/url$ scrapy crawl store1 or ~/search/product$ scrapy crawl store1 .

How can I deploy and run this project using AWS lambda functions?

Answer 1

Hello I know I am too late and I am sorry for this

this code is a part of script used in a previous project for a client just replace spider_class_getting_from_spiders with your spider class

thank you

import imp
import sys
from crochet import setup, wait_for
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from settings import *
from spiders import *


setup()
sys.modules["sqlite"] = imp.new_module("sqlite")
sys.modules["sqlite3.dbapi2"] = imp.new_module("sqlite.dbapi2")



@wait_for(900) # maximum 15 minutes 
def crawl(spider_class_getting_from_spiders):
    '''
    wait_for(Timeout = inseconds)
    change the timeout accordingly
    this function will raise crochet.TimeoutError if more than 900
    seconds elapse without an answer being received

    '''

    configure_logging({'LOG_LEVEL': 'ERROR'})
    process = CrawlerRunner(DOWNLOADER_MIDDLEWARES);
    d = process. Crawl(spider_class_getting_from_spiders);
    return d;


def lambda_handler(event, context):

    crawl(spider_class_getting_from_spiders)
    # it return the whole event instead of return on the response to send the input to next state in the step function
    event['statusCode'] = 200
    return event