[英]Scrapy ModuleNotFoundError: No module named 'MySQLdb'
剛開始使用 Scrapy,我正在嘗試寫入 MySQL 數據庫而不是輸出到 csv。
我在這里找到了代碼: https : //gist.github.com/tzermias/6982723 ,我用它來嘗試完成這項工作,但不幸的是出現了一個我無法理解的錯誤。
這是我的pipelines.py:
class WebsitePipeline(object):
def process_item(self, item, spider):
return item
import MySQLdb.cursors
from twisted.enterprise import adbapi
from scrapy.xlib.pydispatch import dispatcher
from scrapy import signals
from scrapy.utils.project import get_project_settings
from scrapy import log
SETTINGS = get_project_settings()
class MySQLPipeline(object):
@classmethod
def from_crawler(cls, crawler):
return cls(crawler.stats)
def __init__(self, stats):
#Instantiate DB
self.dbpool = adbapi.ConnectionPool ('MySQLdb',
host=SETTINGS['DB_HOST'],
user=SETTINGS['DB_USER'],
passwd=SETTINGS['DB_PASSWD'],
port=SETTINGS['DB_PORT'],
db=SETTINGS['DB_DB'],
charset='utf8',
use_unicode = True,
cursorclass=MySQLdb.cursors.DictCursor
)
self.stats = stats
dispatcher.connect(self.spider_closed, signals.spider_closed)
def spider_closed(self, spider):
""" Cleanup function, called after crawing has finished to close open
objects.
Close ConnectionPool. """
self.dbpool.close()
def process_item(self, item, spider):
query = self.dbpool.runInteraction(self._insert_record, item)
query.addErrback(self._handle_error)
return item
def _insert_record(self, tx, item):
result = tx.execute(
""" INSERT INTO table VALUES (1,2,3)"""
)
if result > 0:
self.stats.inc_value('database/items_added')
def _handle_error(self, e):
log.err(e)
這是我的 settings.py 中的內容:
# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
'Website.pipelines.MySQLPipeline': 300,
}
#Database settings
DB_HOST = 'localhost'
DB_PORT = 3306
DB_USER = 'username'
DB_PASSWD = 'password'
DB_DB = 'scrape'
這是spider.py:
# -*- coding: utf-8 -*-
import scrapy
from scrapy.spiders import SitemapSpider
class WebsitesitemapSpider(SitemapSpider):
name = 'Websitesitemap'
allowed_domains = ['Website.com']
sitemap_urls = ['https://www.Website.com/robots.txt']
def parse(self, response):
yield {response.url}
我一直無法找到我想要做的事情的工作示例,以便能夠找出我出錯的地方,因此感謝任何查看此內容或可能提供幫助的人。
您是否安裝了這些軟件包“MySQLdb,scrapy,twisted”。
否則嘗試使用 PIP 安裝,然后嘗試運行腳本。
你需要在你的 python 環境中安裝 MySQL-python,以及在操作系統上安裝 libmysql。
在 Ubuntu 上,這將通過以下方式實現。
pip install MySQL-python sudo apt-get install libmysql-dev
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.