解析带有Pandas标头问题的脏文本文件

Question

I am trying to parse a text file created back in '99 that is slightly difficult to deal with. The headers are in the first row and are delimited by '^' (the entire file is ^ delimited). The issue is that there are characters that appear to be thrown in (long lines of spaces for example appear to separate the headers from the rest of the data points in the file. (example file located at https://www.chicagofed.org/applications/bhc/bhc-home My example was referencing Q3 1999).

Issues: 1) Too many headers to manually create them and I need to do this for many files that may have new headers as we move forward or backwards throughout the time series 2) I need to recreate the headers from the file and then remove them so that I don't pollute my entire first row with header duplicates. I realize I could probably slice the dataframe [1:] after the fact and just get rid of it, but that's sloppy and i'm sure there's a better way. 3) the unreported fields by company appear to show up as "^^^^^^^^^", which is fine, but will pandas automatically populate NaNs in that scenario?

My attempt below is simply trying to isolate the headers, but i'm really stuck on the larger issue of the way the text file is structured. Any recommendations or obvious easy tricks i'm missing?

from zipfile import ZipFile
import pandas as pd

def main():
    #Driver

    FILENAME_PREFIX = 'bhcf'
    FILE_TYPE = '.txt'
    field_headers = []

    with ZipFile('reg_data.zip', 'r') as zip:

        with zip.open(FILENAME_PREFIX + '9909'+ FILE_TYPE) as qtr_file:
            headers_df = pd.read_csv(qtr_file, sep='^', header=None)

            headers_df = headers_df[:1]
            headers_array = headers_df.values[0]

            parsed_data = pd.read_csv(qtr_file, sep='^',header=headers_array)

Answer 1

I try with the file you linked and one i downloaded i think from 2015:

import pandas as pd
df = pd.read_csv('bhcf9909.txt',sep='^')
first_headers = df.columns.tolist()
df_more_actual = pd.read_csv('bhcf1506.txt',sep='^')
second_headers = df_more_actual.columns.tolist()
print(df.shape)
print(df_more_actual.shape)
# df_more_actual has more columns than first one
# Normalize column names to avoid duplicate columns
df.columns = df.columns.str.upper()
df_more_actual.columns = df_more_actual.columns.str.upper()
new_df = df.append(df_parsed2)
print(new_df.shape)

The final dataframe has the rows of both csv, and the union of columns from them. You can do this for the csv of each quarter and appending it so finally you will have all the rows of them and the union of the columns.