在 Python 中将巨大的镶木地板文件读入 memory 的最有效方法

Question

Ideally, I would like to have the data in a dictionary. I am not even sure if a dictionary is better than a dataframe in this context. After a bit of research, I found the following ways to read a parquet file into memory:

• Pyarrow (Python API of Apache Arrow):

With pyarrow, I can read a parquet file into a pyarrow.Table. I can also read the data into a pyarrow.DictionaryArray. Both are easily convertible into a dataframe, but wouldn't memory consumption double in this case?

• Pandas:

Via pd.read_parquet. The file is read into a dataframe. Again, would a dataframe perform as well as a dictionary?

• parquet-python (pure python, supports read-only):

Supports reading in each row in a parquet as a dictionary. That means I'd have to merge a lot of nano-dictionaries. I am not sure if this is wise.

Answer 1

Most efficient way to read a huge parquet file into memory in Python, you can consider is using the pyarrow library, which provides high-performance, memory-efficient data structures for working with Parquet files.

import pyarrow.parquet as pq

# Read the Parquet file into a Pandas DataFrame
df = pq.read_pandas(path).to_pandas()

# Convert the DataFrame to a NumPy array
data = df.values