How to read text file with two related data sets using Python

Question

I need to parse a fixed width text file which contains weather station's data in this format:

Header row (of particular width and columns)
Data row (of different fixed width and columns)
Data row
.
.
Header row
Data row
Data row
Data row
.
.
Header row
Data row
.

header rows : These rows starts with '#' and contain metadata information about the weather station and a field which tells us how many data lines to read under this header.

Data rows : The data rows contain the actual detailed weather data related to the header present above it.

Sample :

# ID1 A1 B 2 C1
11 20
22 30
# ID2 A1 B 3 C2
23 45
10 17
43 12
# ID1 A3 B1 1 C2
21 32

As we can see, the header rows contain an indicator of how many data rows below are related to it

I want to create a dataframe or table such that I can have this consolidated data which looks something like this:

ID1 A1 B 2 C1 11 20
ID1 A1 B 2 C1 22 30
ID2 A1 B 3 C2 23 45
ID2 A1 B 3 C2 10 17
.
.

please suggest how to go about it.

Answer 1

You can first process the text file and split each rows into a list of their content, then append them into a list as you desire. From there, you can create the dataframe into your desired output:

import pandas as pd

# Read the lines from the text file
with open('test.txt', 'r') as f:
    text_data = f.readlines()

data = [] # The list to append into
current_header = None # Placeholder for the header

# Iterate and fill the list
for row in text_data:
    # Track the current header row
    if row[0] == '#':
        current_header = row[1:].split()
    # Include the tracked header prefix to the row
    else:
        data.append(current_header + row.split())

# Your desired dataframe output
print(pd.DataFrame(data))