从 dataframe 中删除行，直到找到实际的列名

Question

I am reading tabular data from the email in the pandas dataframe. There is no guarantee that column names will contain in the first row.Sometimes data is in the following format. The column names that will always be there are [ID,Name and Year].Sometimes there can be additional columns such as "Age"

dummy1           dummy2     dummy3      dummy4
test_column1 test_column2 test_column3  test_column4
ID     Name        Year                  Age
1      John        Sophomore             20
2      Lisa        Junior                21
3      Ed          Senior                22

Sometimes the column names come in the first row as expected.

ID     Name        Year
1      John        Sophomore
2      Lisa        Junior
3      Ed          Senior

Once I read the HTML table from the email,how can I remove the initial rows that don't contain the column names?["ID","Name","Year"] So in the first case I would need to remove first 2 rows in the dataframe(including column row) and in the second case,i wouldn't have to remove anything.

Also,the column names can be in any sequence,and they can be variable.But these 3 columns will always be there ["ID","Name","Year"] if i do the following,it only works if the dataframe contains only 3 columns ["ID","Name","Year"]

col_index = df.index[(df == ["ID","Name","Year"]).all(1)].item()    # get columns index

df.columns = df.iloc[col_index].to_numpy()   # set valid columns
df = df.iloc[col_index + 1 :]

I should be able to fetch the corresponding column index as long as the row contains any of these 3 columns ["ID","Name","Year"] How can I achieve this? I tried

col_index = df.index[(["ID","Name","Year"] in df).any(1)].item()

But i am getting error

Answer 1

You could stack the dataframe and use isin to find the header row.

IIUC, a small function could work. (personally I'd change this to pass in your file I/O read method and return a dataframe starting at that header row.

#make sure your read method has pd.read..(headers=None)
def find_columns(dataframe,cols) -> list:
    stack_df = dataframe.stack()
    header_row = stack_df[stack_df.isin(cols)].index.get_level_values(0)[0]
    return header_row

---

header_row = find_columns(df,["Age", "Year", "ID", "Name"])

new_df = pd.read_csv(file,skiprows=header_row)

   ID  Name       Year  Age
0   1  John  Sophomore   20
1   2  Lisa     Junior   21
2   3    Ed     Senior   22