python pandas 使用 keras 规范化列，然后拆分为组

Question

Having the following data frame (actual data frame contains multiple strings and numeric columns):

col1    col2
0   A   10
1   A   10
2   B   5
3   B   5

I want to normalize the data based on column values so the result would look like this:

    col1    col2
0   A           0.632456
1   A           0.632456
2   B           0.316228
3   B           0.316228

And then split it to groups to get:

    col1    col2
0   A           0.632456
1   A           0.632456

    col1    col2
0   B           0.316228
1   B           0.316228

Splitting to groups is easy however I'm struggling with the normalization. I've tried using the following code:

from keras.utils import normalize
df = pd.DataFrame({"col1":["A","A","B","B"],"col2":[10,10,5,5]})
normalize(df, axis=0)

But since I have strings it fails, it will work if the values of A and B would be numeric.

Q: How can I normalize the numeric values by columns without dropping the string columns so I can later group by?

Answer 1

When dealing with categorical data, you should be looking at encoding methods such as a OneHotEncoder . It doesn't make sense to try to normalize these columns directly. In this case, you could use a scaler such as MinMaxScaler for the numerical columns (or keras' Normalize ), and then one hot encode the categorical columns as:

from sklearn.preprocessing import MinMaxScaler, OneHotEncoder

sc = MinMaxScaler()
oh = OneHotEncoder()

col2_norm = sc.fit_transform(df.col2.to_numpy()[:,None])
col1_one_hot = oh.fit_transform(df.col1.to_numpy()[:,None]).toarray()

np.concatenate([col1_one_hot, col2_norm], axis=1)
array([[1., 0., 1.],
       [1., 0., 1.],
       [0., 1., 0.],
       [0., 1., 0.]])

If you just want to normalize the categorical column, you can just feed a Series to the scaler, rather than the entire dataframe:

sc = MinMaxScaler()
df['col2'] = sc.fit_transform(df.col2.to_numpy()[:,None])

Or similarly with keras' normalize :

df['col2'] = normalize(df.col2.to_numpy()).squeeze()

---

print(df)

  col1  col2
0    A   1.0
1    A   1.0
2    B   0.0
3    B   0.0    ​