numpy 数组的矢量化“逐层”缩放

Question

I have a numpy array (let's say 100x64x64).

My goal is to scale each 64x64 layer independently and store a scaler for later use.

This is how it can be achieved with a for-loop solution:

scalers_dict={}    
for i in range(X.shape[0]):
            scalers_dict[i] = MinMaxScaler()
            #fitting the scaler
            X[i, :, :] = scalers_dict[i].fit_transform(X[i, :, :])
#saving dict of scalers
joblib.dump(value=scalers_dict,filename="dict_of_scalers.scaler")

My real array is much bigger, and it takes quite a while to iterate through it.

Do you have in mind some more vectorized solution for that, or for-loop is the only way?

Answer 1

If I understand correctly how MinMaxScaler works, it can operate on independent arrays which reduce along axis=0 .

To make this useful for your case, you'd need to transform X into a (64 * 64, 100) array:

s = X.shape
X = np.moveaxis(X, 0, -1).reshape(-1, s[0])

Alternatively, you can write

X = X.reshape(s[0], -1).T

Now you can do the scaling with

M = MinMaxScaler()
X = M.fit_transform(X)

Since the actual fit is computed on the first dimension, all the results will be of size 100. This will broadcast perfectly now that the last dimension is of the same size.

To get the original shape back, invert the original transformation:

X = X.T.reshape(s)

When you are done, M will be a scaler calibrated for 100 features. There is no need for a dictionary here. Remember that a dictionary keyed by a sequence of integers can better be expressed as a list or array, which is what happens here.

Answer 2

IIUC, you can manually scale:

mm, MM = inputs.min(axis=(1,2)), inputs.max(axis=(1,2))

# save these for later use
joblib.dump((mm,MM), 'minmax.joblib')

def scale(inputs, mm, MM):
    return (inputs - mm[:,None,None])/(MM-mm)[:,None,None]

# load pre-saved min & max
mm, MM = joblib.load('minmax.joblib')

# scaled inputs
scale(inputs, mm, MM)