如何在python中读取大型.jl文件

Question

I'm trying to read the following dataset and turn it into a pandas dataframe:
https://www.kaggle.com/marlesson/meli-data-challenge-2020

It is a file with lines with the following format:

{'event_info': '...', 'event_timestamp': '...', 'event_type': '...'}
{'event_info': '...', 'event_timestamp': '...', 'event_type': '...'}
{'event_info': '...', 'event_timestamp': '...', 'event_type': '...'}

I've been trying the following but it takes too long (+60min):

import numpy as np
import pandas as pd
import fileinput
import json

%%time

df = pd.DataFrame()
with fileinput.input(files='/kaggle/input/meli-data-challenge-2020/train_dataset.jl') as file:
    for line in file:
        conv = json.loads(line)
        df = df.append(conv, ignore_index=True)
df.head()

In this code, it reads the file line by line as a string, turns each one of them into json, and then appends it into the dataframe.

Is there any way to turn the dataset into a pandas dataframe faster?

Answer 1

The file I was trying to read was a JSON file with multiple objects. Pandas read_json() supports a lines argument for data like this:

%%time

df = pd.read_json('/kaggle/input/meli-data-challenge-2020/item_data.jl', lines=True)

Output: CPU times: user 14.1 s, sys: 3.31 s, total: 17.4 s
Wall time: 18.6 s