pyspark数据帧中的异常值检测

Question

I'm very new to Spark and Hadoop world. I've started learning these topics on my own from Internet. I wanted to know how we can perform outlier detection in Spark DataFrame given that a DataFrame in Spark is immutable? Is there any Spark package or module which can perform this? I'm using PySpark API for Spark, so I will be highly grateful if someone reply on how this can be done in PySpark. Will highly appreciate if I get a small code for performing outlier detection in Spark DataFrame in PySPark(Pyhton). Thanks a lot in advance!

Answer 1

To my knowledge, there is no a API neither a package that dedicated to detecting outliers as the data itself varies depending on the application. However, there are couple of known methods that all help to identify the outliers. Let's first look at what the term outliers means, it simply refers to the extreme values that fall outside the scope/range of the observations. A good example of how these outliers can be seen is when visualizing the data in a histogram fashion or scatter plot, they can strongly influence the statics and much compress the meaningful data. Or they can be seen as a strong influence on the statistical summary of the data. such as after using the mean or the standard deviations. This certainly will be misleading, the danger will be when we use training data that contain outliers, training will take longer time as the model will be struggling with the out-of-range values, hence we land in a less accurate model and poor result or 'never converging objective measure', ie, comparing the output/scoring of the test and training with respect to training time or some accuracy value range.

Although, it is common to have outliers as an undesirable entities in your data, they still can be sign for anomalies and there their detection itself will be a method to spotting frauds or improving security.

Here are some k own methods for outliers detection (more details can be found in this good article ):

• Extreme Value Analysis,
• Probabilistic and Statistical Models,
• Linear Models: reduce the data dimension,
• Proximity-based Models: mainly using clustering.

For the code, I suggest this good tutorial from mapr. ANd hope this answer helps. Good luck.