如何转换一组值，使每个值更接近平均值，但在 PySpark 中具有类似形状的分布（即减少 stdev）

Question

I hope I've described the job I need to do in the correct terms. Essentially, I need to 'compress' a series of values so that all the values are closer to the mean, but their values should be reduced (or increased) relative to their distance from the mean...

The dataframe looks like this:

>>> df[['population', 'postalCode']].show(10)
+----------+----------+
|population|postalCode|
+----------+----------+
|      1464|     96028|
|       465|     96015|
|       366|     96016|
|      5490|     96101|
|       183|     96068|
|       569|     96009|
|       366|     96054|
|        90|     96119|
|       557|     96006|
|       233|     96116|
+----------+----------+
only showing top 10 rows

>>> df.describe().show()
+-------+------------------+------------------+
|summary|        population|        postalCode|
+-------+------------------+------------------+
|  count|              1082|              1082|
|   mean|23348.511090573014| 93458.60813308688|
| stddev|21825.045923603615|1883.6307236060127|
+-------+------------------+------------------+

The population mean is about right for my purposes, but I need the variance around it to be smaller...

Hope that makes sense, any help performing this job either in pyspark or node.js greatly appreciated.

Answer 1

The general idea is to:

1. translate the mean to zero.
2. rescale to the new standard deviation
3. translate to the desired mean (in this case, the original mean)

In pseudo-code, if your values are stored in the variable x :

x.scaled = new.mean + (x - mean(x)) * new.SD/sd(x)

Or, for the specific case of, say, SD=1000 and no change to the mean:

x.scaled = mean(x) + (x - mean(x)) * 1000/sd(x)