在 Pandas Dataframe 中读取大文件文本文件的最快方法

Question

I have several large files (> 4 gb each). Some of them are in a fixed width format and some are pipe delimited. The files have both numeric and text data. Currently I am using the following approach:

df1 = pd.read_fwf(fwFileName, widths = [2, 3, 5, 2, 16], 
                 names = columnNames, dtype = columnTypes,
                 skiprows = 1, engine = 'c', 
                 keep_default_na = False)
df2 = pd.read_csv(pdFileName, sep = '|', names = columnNames, 
                 dtype = columnTypes, useCols = colNumbers, 
                 skiprows = 1, engine = 'c', 
                 keep_default_na = False)

However, this seems to be slower than for example R's read_fwf (from readr) and fread (from data.table). Can I use some other methods that will help speed up reading these files?

I am working on a big server with several cores so memory is not an issue. I can safely load the whole files into memory. Maybe they are the same thing in this case but my goal is to optimize on time and not on resources.

Update

Based on the comments so far here are a few additional details about the data and my ultimate goal.

• These files are compressed (fixed width are zip and pipe delimited are gzip). Therefore, I am not sure if things like Dask will add value for loading. Will they?
• After loading these files, I plan to apply a computationally expensive function to groups of data. Therefore, I need the whole data. Although the data is sorted by groups, ie first x rows are group 1, next y rows are group 2 and so on. Therefore, forming groups on the fly might be more productive? Is there an efficient way of doing that, given that I don't know how many rows to expect for each group?

Answer 1

Since we are taking time as a metric here, then your memory size is not the main factor we should be looking at, actually on the contrary all methods using lazy loading(less memory and only load objects when needed) are much much faster than loading all data at once in memory, you can check out dask as it provides such lazy read function. https://dask.org/

start_time = time.time() 
data = dask.dataframe.read_csv('rg.csv') 
duration = time.time() - start_time
print(f"Time taken {duration} seconds") # less than a second

But as I said this won't load data in memory, but rather load only portions of data when needed, you could however load it in full using:

data.compute()

If you want to load things faster in memory, then you need to have good computing capabilities in your server, a good candidate that could benefit from such capabilities is ParaText https://github.com/wiseio/paratext You can benchmark ParaText against readcsv using the following code:

import time
import paratext
start_time = time.time() 
df = paratext.load_csv_to_pandas("rg.csv") 
duration = time.time() - start_time
print(f"Time taken {duration} seconds")   

import time
import pandas as pd
start_time = time.time() 
df = pd.read_csv("rg.csv") 
duration = time.time() - start_time
print(f"Time taken {duration} seconds")   

Please Note that results may be worse if you don't have enough compute power to support paraText . You can check out the benchmarks for ParaText loading large files here https://deads.gitbooks.io/paratext-bench/content/results_csv_throughput.html .