识别数据框中的ID列

Question

Is there any way to identify columns such as Account_Number, Employee_ID, Transaction_ID etc type of columns automatically in a data frame which are usually not included in model building? Note that there might be more than one record of the same employee across different dates. In short, how to identify useless columns when they are not unique?

Answer 1

There are several ways to recognize the lease important columns/classes/features, in a dataset. Correlation is one of them. Follow the example below by first downloading this movies dataset from Kaggle.

df = pd.read_csv("tmdb_5000_movies.csv")
df = df[["id", "budget", "popularity", "vote_average"]]
df.head()

This is how the dataframe looks:

    id       budget     popularity  vote_average
0   19995   237000000   150.437577  7.2
1   285     300000000   139.082615  6.9
2   206647  245000000   107.376788  6.3
3   49026   250000000   112.312950  7.6
4   49529   260000000   43.926995   6.1

We are looking for an automatic way of detecting that "id" is a useless column.

Let's find the correlation between each column and the other:

corr_df = pd.DataFrame(columns=list(df.columns))
for col_from in df.columns:
    for col_to in df.columns:
        corr_df.loc[col_from, col_to] = df[col_from].corr(df[col_to])
print(corr_df.head())

Correlation is simply a measure between -1 and 1, numbers close to zero indicate that the two classes are uncorrelated, the further you go from zero, (even in the negative direction) is an indication that the two parameters are coupled in some sense. Observe how id has a very small correlation with budget and popularity

                     id     budget popularity vote_average
id                    1 -0.0893767   0.031202    -0.270595
budget       -0.0893767          1   0.505414    0.0931457
popularity     0.031202   0.505414          1     0.273952
vote_average  -0.270595  0.0931457   0.273952            1

Let's go a little step further and get the absolute value and sum all the correlations, the class with the least correlation score is considered the least useless:

corr_df = corr_df.abs()
corr_df["sum"] = corr_df.sum(axis=0) - 1
print(corr_df.head())

Result:

                     id     budget popularity vote_average       sum
id                    1  0.0893767   0.031202     0.270595  0.391173
budget        0.0893767          1   0.505414    0.0931457  0.687936
popularity     0.031202   0.505414          1     0.273952  0.810568
vote_average   0.270595  0.0931457   0.273952            1  0.637692

Not that there are many issues with this method, for example: if ids are increasing from 0 to N and there is a value that is also increasing amongst the rows with a constant rate, their correlation will be high; moreover, some column X might yield a smaller correlation with column Y than the correlation between Y and id; nevertheless the absolute sum result is good enough in most cases.