Tensorflow分区的csv input_fn

Question

The problem, to sum it up, is that I have the data I want to use for training split into a lot of smaller csvs( feat-01.csv , feat-02.csv , etc). I am trying to feed these to an Estimator , more exactly to do this via some sort of input_fn .

My ideal solution would've been to have some sort of input function that takes a dask.Dataframe (which is pretty much how I generated my data until now) and batch it to the estimator. I tried something along these lines:

import tensorflow as tf
import dask.dataframe as dd

ddf = dd.read_csv('feat-*.csv')
tf.contrib.learn.extract_dask_data(ddf)

However this fails with:

TypeError: Expected `meta` to specify type DataFrame, got type Index

I kinda gave up on this idea due to the lack of documentation about using dask directly, although some docstrings seem to point out that it should be possible. I was thinking about making an input_fn to feed it directly from the csv files, but I found no specific examples about this usecase either.

Being a bit of TF noob, I was wondering what the cleanest method to accomplish is.

Answer 1

UPDATE: After trying fruitlessly to implement it via dask I gave up on the idea both from frustration and because the overhead might be a little to much.

I implemented an input function using tf's queues with pretty good results. Here is the code . Although slightly more complicated than what I had in mind with simply passing dataframes to the the estimator, but doing all the work inside tensorflow seems the most elegant approach.

FINAL UPDATE: Shortly after I posted this question tensorlfow 1.4 was released, and with it the dataset API was officially supported and better documented. If anyone is still interested in this question, I advise you to check out this paragraph from the TF documentation.