iteratively read (tsv) file for Pandas DataFrame

Question

I have some experimental data which looks like this - http://paste2.org/YzJL4e1b (too long to post here). The blocks which are separated by field name lines are different trials of the same experiment - I would like to read everything in a pandas dataframe but have it bin together certain trials (for instance 0,1,6,7 taken together - and 2,3,4,5 taken together in another group). This is because different trials have slightly different conditions and I would like to analyze the results difference between these conditions. I have a list of numbers for different conditions from another file.

Currently I am doing this:

tracker_data = pd.DataFrame
tracker_data = tracker_data.from_csv(bhpath+i+'_wmet.tsv', sep='\t', header=4)
tracker_data['GazePointXLeft'] = tracker_data['GazePointXLeft'].astype(np.float64) 

but this of course just reads everything in one go (including the field name lines) - it would be great if I could nest the blocks somehow which allows me to easily access them via numeric indices...

Do you have any ideas how I could best do this?

Answer 1

You should use read_csv rather than from_csv *:

tracker_data = pd.read_csv(bhpath+i+'_wmet.tsv', sep='\t', header=4)

If you want to join a list of DataFrames like this you could use concat:

trackers = (pd.read_csv(bhpath+i+'_wmet.tsv', sep='\t', header=4) for i in range(?))
df = pd.concat(trackers)

* which I think is deprecated.

Answer 2

I haven't quite got it working, but I think that's because of how I copy/pasted the data. Try this, let me know if it doesn't work.

Using some inspiration from this question

pat = "TimeStamp\tGazePointXLeft\tGazePointYLeft\tValidityLeft\tGazePointXRight\tGazePointYRight\tValidityRight\tGazePointX\tGazePointY\tEvent\n"
with open('rec.txt') as infile:
    header, names, tail = infile.read().partition(pat)

names = names.split()  # get rid of the tabs here
all_data = tail.split(pat)
res = [pd.read_csv(StringIO(x), sep='\t', names=names) for x in all_data]

We read in the whole file so this won't work for huge files, and then partition it based on the known line giving the column names. tail is just a string with the rest of the data so we can split that, again based on the names. There may be a better way than using StringIO, but this should work.

I'm note sure how you want to join the separate blocks together, but this leaves them as a list. You can concat from there however you desire.

For larger files you might want to write a generator to read until you hit the column names and write a new file until you hit them again. Then read those in separately using something like Andy's answer.

A separate question from how to work with the multiple blocks. Assuming you've got the list of Dataframe s, which I've called res , you can use pandas' concat to join them together into a single DataFrame with a MultiIndex (also see the link Andy posted).

In [122]: df = pd.concat(res, axis=1, keys=['a', 'b', 'c'])  # Use whatever makes sense for the keys

In [123]: df.xs('TimeStamp', level=1, axis=1)
Out[123]: 
     a    b    c
0  NaN  NaN  NaN
1  0.0  0.0  0.0
2  3.3  3.3  3.3
3  6.6  6.6  6.6

Answer 3

I ended up doing it iteratively. very very iteratively. Nothing else seems to work.

pat = 'TimeStamp    GazePointXLeft  GazePointYLeft  ValidityLeft    GazePointXRight GazePointYRight ValidityRight   GazePointX  GazePointY  Event'
with open(bhpath+fileid+'_wmet.tsv') as infile:
    eye_data = infile.read().split(pat)
    eye_data = [trial.split('\r\n') for trial in eye_data] # split at '\r'
    for idx, trial in enumerate(eye_data):
        trial = [row.split('\t') for row in trial]
        eye_data[idx] = trial