從Songkick檢索數據時出現內存錯誤

Question

我已經建立了一個刮板，使用其api從Songkick檢索音樂會數據。 但是，從這些藝術家那里檢索所有數據需要花費大量時間。 抓取大約15小時后，腳本仍在運行，但JSON文件不再更改。 我中斷了腳本，並檢查了是否可以使用TinyDB訪問數據。 不幸的是，我收到以下錯誤。 有人知道為什么會這樣嗎？

錯誤：

('cannot fetch url', 'http://api.songkick.com/api/3.0/artists/8689004/gigography.json?apikey=###########&min_date=2015-04-25&max_date=2017-03-01')
8961344


Traceback (most recent call last):
  File "C:\Users\rmlj\Dropbox\Data\concerts.py", line 42, in <module>
    load_events()
  File "C:\Users\rmlj\Dropbox\Data\concerts.py", line 27, in load_events
    print(artist)
  File "C:\Python27\lib\idlelib\PyShell.py", line 1356, in write
    return self.shell.write(s, self.tags)
KeyboardInterrupt

>>> mydat = db.all()

Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    mydat = db.all()
  File "C:\Python27\lib\site-packages\tinydb\database.py", line 304, in all
    return list(itervalues(self._read()))
  File "C:\Python27\lib\site-packages\tinydb\database.py", line 277, in _read
    return self._storage.read()
  File "C:\Python27\lib\site-packages\tinydb\database.py", line 31, in read
    raw_data = (self._storage.read() or {})[self._table_name]
  File "C:\Python27\lib\site-packages\tinydb\storages.py", line 105, in read
    return json.load(self._handle)
  File "C:\Python27\lib\json\__init__.py", line 287, in load
    return loads(fp.read(),
MemoryError

在下面你可以找到我的腳本

 import urllib2
import requests
import json
import csv
import codecs


from tinydb import TinyDB, Query
db = TinyDB('events.json')


def load_events():
        MIN_DATE = "2015-04-25"
        MAX_DATE = "2017-03-01"
        API_KEY= "###############"
        with open('artistid.txt', 'r') as f:
            for a in f: 
                artist = a.strip() 
                print(artist)
                url_base = 'http://api.songkick.com/api/3.0/artists/{}/gigography.json?apikey={}&min_date={}&max_date={}'
                url = url_base.format(artist, API_KEY, MIN_DATE, MAX_DATE)
                # url = u'http://api.songkick.com/api/3.0/search/artists.json?query='+artist+'&apikey=WBmvXDarTCEfqq7h'
                try:
                  r = requests.get(url)
                  resp = r.json()
                  if(resp['resultsPage']['totalEntries']):
                    results = resp['resultsPage']['results']['event']
                    for x in results:
                        print(x)
                        db.insert(x)
                except:
                    print('cannot fetch url',url);

load_events()
db.close()
print ("End of script")    

Answer 1

MemoryError是內置的Python異常（ https://docs.python.org/3.6/library/exceptions.html#MemoryError ），因此該進程似乎內存不足，並且與Songkick並沒有真正的關系。

這個問題可能包含您需要調試的信息： 如何在Python中調試MemoryError？ 跟蹤內存使用情況的工具？