从 dataframe 中删除行，直到找到实际的列名

Question

I am reading tabular data from the email in the pandas dataframe. There is no guarantee that column names will contain in the first row.Sometimes data is in the following format.The actual column names are [ID,Name and Year]

dummy1           dummy2     dummy3
test_column1 test_column2 test_column3
ID     Name        Year
1      John        Sophomore
2      Lisa        Junior
3      Ed          Senior

Sometimes the column names come in the first row as expected.

ID     Name        Year
1      John        Sophomore
2      Lisa        Junior
3      Ed          Senior

Once I read the HTML table from the email,how I remove the initial rows that don't contain the column names?So in the first case I would need to remove first 2 rows in the dataframe(including column row) and in the second case,i wouldn't have to remove anything.

Also,the column names can be in any sequence. basically,I want to do in following

1.check whether once of the column names contains in one of the rows in dataframe
2.Remove the rows above
if "ID" in row:
    remove the above rows

How can I achieve this?

Answer 1

Ugly but effective quick try:

id_name = df.columns[0]
df_clean = df[(df[id_name] == 'ID') | (df[id_name].dtype == 'int64')]

Answer 2

You can first get index of valid columns and then filter and set accordingly.

df = pd.read_csv("d.csv",sep='\s+', header=None)

col_index = df.index[(df == ["ID","Name","Year"]).all(1)].item()    # get columns index

df.columns = df.iloc[col_index].to_numpy()   # set valid columns
df = df.iloc[col_index + 1 :]                # filter data
df
  ID  Name       Year
3  1  John  Sophomore
4  2  Lisa     Junior
5  3    Ed     Senior

or

If you want to se ID as index

df = df.iloc[col_index + 1 :].set_index('ID')
df
    Name       Year
ID
1   John  Sophomore
2   Lisa     Junior
3     Ed     Senior