在python中使用数字数据读取csv的有效方法

Question

I try to convert a code writen in Matlab into python. I'm trying to read dat file (it's a csv file). that file has about 30 columns and thousands of rows containing (only!) decimal number data (in Matlab it was read into double matrix). I'm asking for the fastest way to read the dat file and the most similar object/array/... to save the data into.

I tried to read the file in both of the following ways:

my_data1 = numpy.genfromtxt('FileName.dat', delimiter=',' )
my_data2 = pd.read_csv('FileName.dat',delimiter=',')

Is there any better option?

Answer 1

pd.read_csv is pretty efficient as it is. To make it faster, you can use try to use multiple cores to load your data in parallel. Here is some code example where I used joblib when I needed to make data loading with pd.read_csv and processing of that data faster.

from os import listdir
from os.path import dirname, abspath, isfile, join
import pandas as pd
import sys
import time
from datetime import datetime
# Multi-threading
from joblib import Parallel, delayed
import multiprocessing
# Garbage collector
import gc

# Number of cores
TOTAL_NUM_CORES = multiprocessing.cpu_count()
# Path of this script's file
DATA_PATH = 'D:\\'
# Path to save the processed files
TARGET_PATH = 'C:\\'

def read_and_convert(f,num_files):
    #global i
    # Read the file
    dataframe = pd.read_csv(DATA_PATH + f, low_memory=False, header=None, names=['Symbol', 'Date_Time', 'Bid', 'Ask'], index_col=1, parse_dates=True)
    # Process the data
    data_ask_bid = process_data(dataframe)
    # Store processed data in target folder
    data_ask_bid.to_csv(TARGET_PATH + f)
    print(f)
    # Garbage collector. I needed to use this, otherwise my memory would get full after a few files, but you might not need it.
    gc.collect()

def main():
    # Counter for converted files
    global i
    i = 0
    start_time = time.time()
    # Get the paths for all the data files
    files_names = [f for f in listdir(DATA_PATH) if isfile(join(DATA_PATH, f))]

    # Load and process files in parallel
    Parallel(n_jobs=TOTAL_NUM_CORES)(delayed(read_and_convert)(f,len(files_names)) for f in files_names)
    # for f in files_names: read_and_convert(f,len(files_names)) # non-parallel
    print("\nTook %s seconds." % (time.time() - start_time))

if __name__ == "__main__":
    main()