在 python 中处理大型 JSON 数据

Question

My JSON (~500mb) file has multiple JSON objetcs, actually i just need to use the "customer_id" colunm. When i execute the code below, it gives memory error.

with open('online_pageviews.json') as f:
     online_pageviews = pd.DataFrame(json.loads(line) for line in f)

Here is a example of an JSON object in "online_pageviews.json"

{
"date": "2018-08-01",
"visitor_id": "3832636531373538373137373",
"deviceType": "mobile",
"pageType": "product",
"category_id": "6365313034",
"on_product_id": "323239323839626",
"customer_id": "33343163316564313264"
}

Is there a way to only use the "customer_id" column? What can i do to load this file?

Answer 1

You should be able to do this if you manage the amount of data you actually have floating around. Since you only need the customer ID don't bother loading any of the other data into your dataframe.

customer_id_array = []
with open('online_pageviews.json') as f:
    for line in f:
        customer_id_array.append(json.loads(line)['customer_id'])
online_pageviews = pd.DataFrame(customer_id_array,columns = ['customer_id'])

This way can you significantly cut down on how much extra memory you were previously using.

(Im not sure if your system will be able to handle this as customer_id_array can still get pretty big but it should be much better than before. If it cannot you may need to look for some online options for renting systems with more memory.)