熊猫中的并行read_table

Question

Is there a way to parallelize an invocation of read_table()? In my case it's CPU bound due to date parsing. I don't see any way to achieve that from reading the docs. The only thing that comes to mind is splitting the input file, calling read_table in parallel and then concatenating the dataframes.

Answer 1

This will read CSV files in parallel and concatenate them. The annoying bit is that it won't handle numpy types, so it can't parse dates. I have been struggling with the same problem, but so far it seems that libraries such as execnet can't handle types that are not built-in. That's why I turn DataFrames into json before sending. It strips the types to basic Python ones.

Edit: If you need to parse dates, maybe a more sensible approach would be to remotely read the CSV files, parse the dates and save them as pickle to the hard drive. Then you could read the pickle files in the main process and concatenate them. I haven't tried that to see if it would lead to a gain in performance.

remote_read_csv.py

import cPickle as pickle

if __name__ == '__channelexec__':
    reader = pickle.loads(channel.receive())

    for filename in channel:
        channel.send(reader(filename).to_json())

This below makes use of the module above. I tested it in IPython.

from pandas import DataFrame, concat, read_csv, read_json
from numpy import random
import execnet
import remote_read_csv
import cPickle as pickle
import itertools
import psutil

### Create dummy data and save to CSV

def rdf():
    return DataFrame((random.rand(4, 3) * 100).astype(int))

d1 = rdf()
d2 = rdf()
d3 = rdf()

dfsl = [d1, d2, d3]
names = 'd1.csv d2.csv d3.csv'.split()
for i in range(3):
    dfsl[i].to_csv(names[i])

### Read CSV files in separate threads then concatenate

reader = pickle.dumps(read_csv)

def set_gateways(remote_module, *channel_sends):
    gateways = []
    channels = []
    for i in range(psutil.NUM_CPUS):
        gateways.append(execnet.makegateway())
        channels.append(gateways[i].remote_exec(remote_module))
        for send in channel_sends:
            channels[i].send(send)
    return (gateways, channels)

def para_read(names):
    gateways, channels = set_gateways(remote_read_csv, reader)
    mch = execnet.MultiChannel(channels)
    queue = mch.make_receive_queue()
    channel_ring = itertools.cycle(mch)
    for f in names:
        channel = channel_ring.next()
        channel.send(f)
    dfs = []
    for i in range(len(names)):
        channel, df = queue.get()
        dfs.append(df)

    [gw.exit() for gw in gateways]
    return concat([read_json(i) for i in dfs], keys=names)

para_read(names)