Construct DataFrame from values between a sequence of line in txt file

Question

I'm having difficulty in getting values from unstructured lines in a txt file. Alpha and Beta are the key of my data refrence and X_1,X_2,X_3 are the variable that I need to get from the file (just simplification, in real data there are 192 variables).

I want to extract the last value (separated by whitespace) of each X_n (n=1,2,3) into a dict of every pair of Alpha and Beta value. More or less to get a 3D panel like this one. desirable dataframe

The lines of string in file.txt file that look like this (after parsing):

Alpha = 180
Beta = 0
X_1 3.34 5
X_3 4.34 7
Alpha = 180
Beta = 10
X_1 4.23 2
X_2 3.23 1 
Alpha = 180
Beta = 20
X_2 3.23 9
.
.
.
.
Alpha = 180
Beta = 90
X_1 7.23 3
X_2 9.14 3
X_3 5.91 7 
Alpha = 170
Beta = 0
X_1 7.63 3
X_2 4.84 2
X_3 8.01 8 
.
. 
(and so on)

My goal is create 3D panel like below

Alpha Beta x_1 x_2 x_3
180    0     5   0   7
180    10    2   1   0
180    20    0   0   9

180    90    3   3   7
170     0    3   2   8

as far I've tried. I can get the value of X_1,X_2 X_3 by using regex

readings = []
with open('file.txt') as inputfile:
    for line in inputfile:
    readings.append(line.strip())

x_1_list =[]
for r in readings:
    if re.search('x_1,r')
        c = re.split(r'\s+',r)[-1]
        x_1_list.append(c)
    else:
        x_1_list.append(0.0)

However, I wasn't able to use for loop this function for every Alpha and Beta value.

Any suggestions?

Answer 1

I would suggest parsing the input file as groups of ['Alpha', 'Beta', 'X_1', 'X_2', 'X_3'] values, ie group by group, rather than line by line.

Below is the working code. I hope it explains itself, but please comment if it needs clarification.

KEYS = ['Alpha', 'Beta', 'X_1', 'X_2', 'X_3']
GROUP_START_MARKER = KEYS[0]


def parse_group(handle, line):
    value_dict = {}
    assert line.startswith(GROUP_START_MARKER)
    alpha_value = line.split(' = ')[1]
    value_dict[GROUP_START_MARKER] = alpha_value
    line = handle.readline().strip()
    assert line.startswith('Beta')
    beta_value = line.split(' = ')[1]
    value_dict['Beta'] = beta_value
    readings = []
    while True:
        line = handle.readline().strip()
        if line.startswith(GROUP_START_MARKER):
            break
        if not line:
            break
        key, _, value = line.split()
        value_dict[key] = value
    return value_dict, line


def parse_file(filename):
    value_list = []
    with open(filename) as inputfile:
        line = inputfile.readline().strip()
        while True:
            if line.startswith(GROUP_START_MARKER):
                value_dict, line = parse_group(inputfile, line)
                if not line:
                    return
                yield value_dict


it = parse_file('file.txt')
print '\t'.join(KEYS)
for value_dict in it:
    print '\t'.join(map(str, [value_dict.get(key, 0.0) for key in KEYS]))