在Python中本地存储数据

Question

I'm running a Scrapy project and I'm looking for the best method for storing already scraped data locally. Currently I'm using AnyDBM but i keep getting the following error after a while of running:

bsddb.db.DBRunRecoveryError: (-30973, 'DB_RUNRECOVERY: Fatal error, run database recovery -- PANIC: fatal region error detected; run recovery')

It my be something I'm doing wrong as I'm pretty new to Python, but i was wondering if there is a better solution other than Anydbm anyway.

I'm storing numeric ID's of the pages I have crawled, and will be storing around 500,000 records with plans for a possible 3-4 million for future projects.

Is AnyDBM what i should be sticking with or should i change to something more suitable for the job.

Answer 1

看起来非常适合sqlite ，已经是Python标准库的一部分。

Answer 2

By default python comes with sqlite3 which is a very good data base system.

Here is a pretty good tutorial on it. To put the table in memory use:

conn = sqlite3.connect(":memory:")
conn.isolation_level = None
cur = conn.cursor()