Storing values from binary output into list/dictionary/tuple (Python)

Question

Just after some assistance as I am new to python. I have a file which is an output file that is constantly written to with data by an external process. I'm looking to poll this file for an 'x' period of time and print the output to screen and/or store it in a list/dictionary/tuple (unsure on what is appropriate in python terms).

This is what I have at this stage:

timeout = time.time() + args.duration
with open('/dev/input/accel', 'rb') as o:
    while time.time() < timeout:
        data = o.read(16)
        sec, microsec, valtype, axis, axisval = struct.unpack(
            'LLHHi', data)
        tampertime = float(str(sec) + '.' + str(microsec))

        if valtype == 3:
            print(tampertime, axis, axisval)

And the output is this (from the structure):

secs.microsecs, axis, axis value
---------------------------
1478574443.4219799, 0, 4080
1478574443.4219799, 1, 1168
1478574443.4219799, 2, -15408
1478574443.542166, 0, 4016
1478574443.542166, 1, 1104
1478574443.6621571, 0, 4022
1478574443.6621571, 1, 1120
1478574443.6621571, 2, -15404
1478574443.7821031, 0, 4016
1478574443.7821031, 1, 1216
1478574443.7821031, 2, -15430
1478574443.9019749, 0, 4022
1478574443.9019749, 1, 1152
1478574444.2220099, 1, 1148
1478574444.2220099, 2, -15344

Note: I have just added the commas in for easier reading.

It prints out other values as well but I only want the figures where valtype == 3 (which is a column not shown in the output above).

To quickly explain the three columns, the first is time (epoch/unix), second is a 'key' and third is the value associated to the key. You will notice that if there has been no change in the key value, it simply omits it from the output (I am unable to fix this, it's just how the output file handles the values).

So using the output above, I am looking for this output format.

secs.microsecs, 0, 1, 2
----------------------------------------
1478574443.4219799, 4080, 1168, -15408*
1478574443.542166, 4016, 1104, -15408*
1478574443.6621571, 4022, 1120, -15404
1478574443.7821031, 4016, 1216, -15430*
1478574443.9019749, 4022*, 1152, -15430*
1478574444.2220099, 4022*, 1148, -15344

You'll notice the values which I have placed the asterisks next to actually correspond to their value previously (above). ie no value was actually provided for that time stamp so it should just use the previous value that was stored.

I have done a bit of searching and thought about making a structure but I haven't had any luck.

Any help would be greatly appreciated.

Answer 1

You can use defaultdict for this:

from collections import defaultdict
d1=defaultdict(dict)
timeout = time.time() + args.duration
with open('/dev/input/accel', 'rb') as o:
    while time.time() < timeout:
        data = o.read(16)
        sec, microsec, valtype, axis, axisval = struct.unpack(
            'LLHHi', data)
        tampertime = float(str(sec) + '.' + str(microsec))
        if valtype == 3:
            d1[tampertime][axis] = axisval # dict with all values

tmp = {0: 0, 1: 0, 2:0}
for k in sorted(d1.keys()): #iterate through dict and customize output format
    for j in range(3):
        if d1[k].get(j) is None:
            d1[k][j] = tmp[j]
    tmp = d1[k]
    print(k, d1[k])

Output:

1478574443.42198 {0: 4080, 1: 1168, 2: -15408}
1478574443.542166 {0: 4016, 1: 1104, 2: -15408}
1478574443.662157 {0: 4022, 1: 1120, 2: -15404}
1478574443.782103 {0: 4016, 1: 1216, 2: -15430}
1478574443.901975 {0: 4022, 1: 1152, 2: -15430}
1478574444.22201 {0: 4022, 1: 1148, 2: -15344}

Answer 2

Just answering my own question. Because I wanted the values to be printed out after each execution (once retrieved x, y, and z values) I used this solution. I noticed that in my binary file there was a flush value that was passed which I could poll and check to trigger a print.

Here's basically my complete code:

class Sample:

    def __init__(self):
        self.x = None
        self.y = None
        self.z = None
        self.ts_sec = None
        self.ts_usec = None

    def print_val(self):
        print("{0}.{1},{2},{3},{4}".format(
            self.ts_sec, self.ts_usec, self.x, self.y, self.z))


def DisplaySamples(samples):
    for s in samples:
    s.print_val()


def UpdateSample(data, sample):
    flush = False
    sec, usec, valtype, axis, axisval = struct.unpack('LLHHi', data)

    sample.ts_sec = sec
    sample.ts_usec = usec
    if valtype == 3:
        if axis == 0:
            sample.x = axisval
        if axis == 1:
            sample.y = axisval
        if axis == 2:
            sample.z = axisval
    if valtype == 0:
        flush = True

    return sample, flush


def main():
    with open('/dev/input/accel', 'rb') as o:
        sample = Sample()
        while time.time() < timeout:
            data = o.read(16)

            sample, flush = UpdateSample(data, sample)

            if flush:
                DisplaySamples([sample])
    o.close()

Example of the output, csv formatted.

1478733511.251977,2896,-367,None
1478733511.372184,2768,-343,-15648
1478733511.492175,2752,-337,-15776
1478733511.612123,2888,-408,-15648
1478733511.732160,2888,-408,-15728
1478733511.852204,2868,-320,-15608
1478733511.972208,2838,-384,-15624
1478733512.92151,2819,-388,-15624
1478733512.212153,2819,-370,-15856
1478733512.332126,2888,-370,-15664
1478733512.452203,2860,-370,-15776
1478733512.572205,2882,-345,-15608
1478733512.692160,2768,-350,-15760
1478733512.812114,2880,-350,-15760
1478733512.932126,2768,-416,-15696
1478733513.52211,2840,-444,-15690