Use Python zip to save data in separate columns from a binary file

Question

I have the following data file :

0.0 2453.4645   4906.929    7360.3935   9813.858    12267.3225  14720.787   17174.2515  19627.716   22081.1805  24534.645   26988.1095  29441.574   31895.0385  34348.503   36801.9675

in BINARY form.

I need to read this into 8 lists a,b,c,d,e,f,g,h with 3 elements each, . ie I need the elements 1-8 saved in each of these variables, and then 9-16 , and so on.

I have the following code:

 # Python code to read binary data

from struct import *
import numpy as np

readfile = open('bigdata.dat')

readfile_data = readfile.read()

type(readfile_data)

a = len(readfile_data)

print a

e = unpack('18d',readfile_data[0:8*18])

field_names = ('a','b','c','d','e','f')

hg = dict(zip(field_names,e))

print hg

What I get is a dict with an element for each dictionary value:

{'a': 0.0, 'c': 4906.929, 'b': 2453.4645, 'e': 9813.858, 'd': 7360.3935, 'f': 12267.3225}

How can I do this in Python (preferably 2.7, but 3 is also welcome)? I assume I have to loop over these dictionary field names over the entire list, but I do not know how..

Answer 1

Once you have the data in list form you can do something like this

from collections import defaultdict

data = [[0.0, 2453.4645, 4906.929, 7360.3935,
        9813.858, 12267.3225, 14720.787,
        17174.2515], [19627.716, 22081.1805,
        24534.645, 26988.1095, 29441.574,
        31895.0385, 34348.503, 36801.9675]]

hg = defaultdict(list)
field_names = ('a','b','c','d','e','f')

for row in data:
    for field_name, datum in zip(field_names, row):
        hg[field_name].append(datum)

print hg

which outputs

defaultdict(<type 'list'>, {'a': [0.0, 19627.716], 'c': [4906.929, 24534.645], 'b': [2453.4645, 22081.1805], 'e': [9813.858, 29441.574], 'd': [7360.3935, 26988.1095], 'f': [12267.3225, 31895.0385]})

Answer 2

The itertools module has an islice() function which may help you:

>>> s = "abcdefghijklmnopqrstuvwxyz"
>>> import itertools
>>> for val in itertools.islice(s, 0, None, 8):
...   print val
...
a
i
q
y
>>> for val in itertools.islice(s, 1, None, 8):
...   print val
...
b
j
r
z
>>> for val in itertools.islice(s, 2, None, 8):
...   print val
...
c
k
s

So for your problem, you might do:

import itertools
a = [item for item in itertools.islice(e, 0, None, 8)]
b = [item for item in itertools.islice(e, 1, None, 8)]
c = [item for item in itertools.islice(e, 2, None, 8)]

and so on. Or, better yet:

columns = []
for n in range(8):
    columns.append([item for item in itertools.islice(e, n, None, 8)])

Hope this helps!

PS Here's the documentation for islice . There are plenty of other useful tools in the itertools module: take a look!