Scikit-Learn 自定义输入器，在平均值附近具有随机值

Question

I want to create a Custom Imputer to replace NaN values in my data with a random value in the range of mean - std and mean + std for the column where the NaN value is in.

This is the code for the Imputer i have so far:

class GroupImputer(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        X = check_array(X, force_all_finite=False)
        self.means = np.nanmean(X, axis=0)
        self.stds = np.nanstd(X, axis=0)
        return self

    def transform(self, X, y=None):
        check_is_fitted(self, 'means')
        check_is_fitted(self, 'stds')
        X = check_array(X, force_all_finite=False)
        # how do i apply to each row of the data?
        return 0

The self.means contains a list of the means for each column.

The self.stds contains a list of all stds for each column.

How do i apply a random value between mean - std and mean + std for each NaN in a row of the data?

Do i have to iterate through the data? ( for row in X: ) and pick the right mean and std according to the column index? Or is there a method which will do this?

Answer 1

No, you don't have to iterate through the data, assume the number of rows and the number of columns of your data are 5 and 4, respectively

num_rows,num_cols = 5,4

# just fake two arrays of column means and stds
column_means = np.random.uniform(1,8,num_cols)
column_stds = np.random.rand(num_cols)

disp = np.random.uniform(column_means-column_stds,column_means+column_stds, size=(num_rows,num_cols))

The array disp is something like

array([[6.29377845, 6.56185572, 5.32590954, 2.14719305],
       [6.36648777, 6.97781432, 4.89773801, 2.21909144],
       [5.38109603, 6.70649396, 5.50100582, 2.26518757],
       [5.59764259, 6.90297057, 5.65199988, 2.25340505],
       [5.80928963, 6.4976407 , 5.23792109, 1.99580784]])

in which each column of this array is uniformly sampled from the range (the column mean - the column std, the column mean + the column std) . Therefore, the NaN entries of the original array can be replaced with the entries of disp .

Answer 2

No, there is a better option then iterate through the data. You can create a uniformly random array (between the desired bounds) with the same shape) and replace every NaN value at index i with the random value at the same index.

higher_bound = self.means + self.stds
lower_bound = self.means - self.stds
random_values = numpy.random.uniform(low=lower_bound, high=higher_bound , size=X.shape) #uniformly random array with the same shape
nan_mask = np.isnan(X) #indicates where is nan
X = np.where(nan_mask, random_values, X) #takes from random_values where nan_mask is true, else takes  from original array