Python耗盡內存

Question

我有以下程序。 當我運行它時，我收到了Memory Error ，特別是在Fpred = F.predict(A) （請參見下文）

import json
data = []
with open('yelp_data.json') as f:
    for line in f:
        data.append(json.loads(line))
star = []
for i in range(len(data)):
    star.append(data[i].values()[10])

attributes = []
for i in range(len(data)):
    attributes.append(data[i].values()[12])


def flatten_dict(dd, separator=' ', prefix=''):
    return { prefix + separator + k if prefix else k : v
         for kk, vv in dd.items()
         for k, v in flatten_dict(vv, separator, kk).items()
         } if isinstance(dd, dict) else { prefix : dd }

flatten_attr = list(flatten_dict(attributes[i], separator = ' ', prefix = '') for i in range(len(attributes)))


from sklearn.feature_extraction import DictVectorizer
v = DictVectorizer(sparse = False)
X = v.fit_transform(flatten_attr)

from sklearn.feature_extraction.text import TfidfTransformer
Transformer = TfidfTransformer()
A = Transformer.fit_transform(X)

from sklearn.linear_model import LinearRegression
from sklearn.cross_validation import train_test_split

from sklearn.neighbors import KNeighborsRegressor
from sklearn.cross_validation import KFold

F = KNeighborsRegressor(n_neighbors = 27)

Ffit = F.fit(A, star)
Fpred = F.predict(A)
Score = F.score(A, star)
print(Score)

我的json文件看起來像這樣 -

{"business_id": "vcNAWiLM4dR7D2nwwJ7nCA", "full_address": "4840 E Indian School Rd\nSte 101\nPhoenix, AZ 85018", "hours": {"Tuesday": {"close": "17:00", "open": "08:00"}, "Friday": {"close": "17:00", "open": "08:00"}, "Monday": {"close": "17:00", "open": "08:00"}, "Wednesday": {"close": "17:00", "open": "08:00"}, "Thursday": {"close": "17:00", "open": "08:00"}}, "open": true, "categories": ["Doctors", "Health & Medical"], "city": "Phoenix", "review_count": 7, "name": "Eric Goldberg, MD", "neighborhoods": [], "longitude": -111.98375799999999, "state": "AZ", "stars": 3.5, "latitude": 33.499313000000001, "attributes": {"By Appointment Only": true}, "type": "business"}
{"business_id": "JwUE5GmEO-sH1FuwJgKBlQ", "full_address": "6162 US Highway 51\nDe Forest, WI 53532", "hours": {}, "open": true, "categories": ["Restaurants"], "city": "De Forest", "review_count": 26, "name": "Pine Cone Restaurant", "neighborhoods": [], "longitude": -89.335843999999994, "state": "WI", "stars": 4.0, "latitude": 43.238892999999997, "attributes": {"Take-out": true, "Good For": {"dessert": false, "latenight": false, "lunch": true, "dinner": false, "breakfast": false, "brunch": false}, "Caters": false, "Noise Level": "average", "Takes Reservations": false, "Delivery": false, "Ambience": {"romantic": false, "intimate": false, "touristy": false, "hipster": false, "divey": false, "classy": false, "trendy": false, "upscale": false, "casual": false}, "Parking": {"garage": false, "street": false, "validated": false, "lot": true, "valet": false}, "Has TV": true, "Outdoor Seating": false, "Attire": "casual", "Alcohol": "none", "Waiter Service": true, "Accepts Credit Cards": true, "Good for Kids": true, "Good For Groups": true, "Price Range": 1}, "type": "business"}

$ls -l yelp_data.json

顯示文件大小為33524921

我能做的更糟糕的是在不同的文件中提取所需的數據並將其導入到該程序中？ 改進這個程序以使其更有效地運行會有什么好處？ 謝謝！！

Answer 1

不是性能/內存相關，但您可以替換：

for i in range(len(data)):
    star.append(data[i].values()[10])

通過：

for item in data:
    star.append(item.values()[10])

data是一個list ，它是可迭代的。 https://docs.python.org/3/library/stdtypes.html#list

同樣在Python 3中，索引dict值不再有效，您將最終得到：

    star.append(data[i].values()[10])
TypeError: 'dict_values' object does not support indexing

由於data中的項是json dicts，您可能希望按名稱搜索屬性，而不是依賴屬性索引：

for item in data:
    star.append(item['thekeyyourelookingfor'])

然后讓它成為單行：

star = [item['thekeyyourelookingfor'] for item in data]

編輯 ：實際上，因為json.loads將JSON字符串讀入字典，所以順序或屬性是任意的，因此當您通過索引訪問它們時，很可能最終會得到與您正在查找的屬性不同的屬性。 我想在這里你想讀stars 。 我甚至猜測這就是你的代碼失敗的原因，因為你提供了他不期望的sklearn輸入。