在python中使用數字數據讀取csv的有效方法

Question

我嘗試將在Matlab中編寫的代碼轉換為python。 我正在嘗試讀取dat文件（這是一個csv文件）。 該文件大約有30列和數千行，其中包含（僅！）十進制數數據（在Matlab中已被讀取為雙矩陣）。 我正在尋求最快的方式來讀取dat文件以及最相似的object / array / ...以將數據保存到其中。

我嘗試通過以下兩種方式讀取文件：

my_data1 = numpy.genfromtxt('FileName.dat', delimiter=',' )
my_data2 = pd.read_csv('FileName.dat',delimiter=',')

有沒有更好的選擇？

Answer 1

pd.read_csv確實非常有效。 為了使其更快，您可以嘗試使用多個內核並行加載數據。 這里是我使用了一些代碼示例joblib當我需要使數據加載與pd.read_csv的數據更快和處理。

from os import listdir
from os.path import dirname, abspath, isfile, join
import pandas as pd
import sys
import time
from datetime import datetime
# Multi-threading
from joblib import Parallel, delayed
import multiprocessing
# Garbage collector
import gc

# Number of cores
TOTAL_NUM_CORES = multiprocessing.cpu_count()
# Path of this script's file
DATA_PATH = 'D:\\'
# Path to save the processed files
TARGET_PATH = 'C:\\'

def read_and_convert(f,num_files):
    #global i
    # Read the file
    dataframe = pd.read_csv(DATA_PATH + f, low_memory=False, header=None, names=['Symbol', 'Date_Time', 'Bid', 'Ask'], index_col=1, parse_dates=True)
    # Process the data
    data_ask_bid = process_data(dataframe)
    # Store processed data in target folder
    data_ask_bid.to_csv(TARGET_PATH + f)
    print(f)
    # Garbage collector. I needed to use this, otherwise my memory would get full after a few files, but you might not need it.
    gc.collect()

def main():
    # Counter for converted files
    global i
    i = 0
    start_time = time.time()
    # Get the paths for all the data files
    files_names = [f for f in listdir(DATA_PATH) if isfile(join(DATA_PATH, f))]

    # Load and process files in parallel
    Parallel(n_jobs=TOTAL_NUM_CORES)(delayed(read_and_convert)(f,len(files_names)) for f in files_names)
    # for f in files_names: read_and_convert(f,len(files_names)) # non-parallel
    print("\nTook %s seconds." % (time.time() - start_time))

if __name__ == "__main__":
    main()