我应该将这些数据点视为异常值吗？

Question

This question was migrated from Stack Overflow because it can be answered on Cross Validated. Migrated 5 days ago .

Currently, I am building my analytics portfolio as part of the Google Data Analytics course. I chose the option to analyze Divvy Bike Sharing data for the year 2021. But now I'm currently stuck in the part where I need to identify outliers in the dataset. I'm focusing on the 'ride_length' column which shows the duration of each ride and I'm using two methods which are:

1. IQR (data points that fall below 25th or above 75th percentile are outliers)
2. 1% and 99% rule (data points that fall below 1% percentile or above 99% percentile are outliers)

Note: the ride_length column is counted in minutes

A) IQR METHOD

The first method that I use to detect outliers is the IQR proximity rule (The data points which fall below the 25th percentile or above the 75th percentile are outliers). Here's the code:

lower_bound_iqr <- quantile(df_2021_test$ride_length, 0.25) upper_bound_iqr <- quantile(df_2021_test$ride_length, 0.75)

lower_bound_iqr
25% 
6.98 

upper_bound_iqr
75% 
20.98 

Key takeaways:

1. ride_length that falls below 6.98 minutes is considered an outlier
2. ride_length that falls above 20.98 minutes is considered an outlier

Then I count the percentages of outliers in the data:

outliers_iqr <- which(df_2021_test$ride_length < lower_bound_iqr | df_2021_test$ride_length > upper_bound_iqr)
(count(df_2021_test[outliers_iqr, ]) / count(df_2021_test)) * 100 

         n
1 49.89316

The result is that 49.89 % of data are considered outliers . I think this is too much data to exclude for the analysis to begin as it will reduce the accuracy of the analysis. Or am I wrong? Therefore I move to the second method

B) 1% and 99% Percentile Rule

This method state that data points that are far from the 99% percentile and less than 1% percentile are considered an outlier. Here's the code:

lower_bound <- quantile(df_2021_test$ride_length, 0.01) upper_bound <- quantile(df_2021_test$ride_length, 0.99)

lower_bound
1% 
1.82 
upper_bound
99% 
115.63 

Key takeaways:

1. ride_length that falls below 1.82 minutes are considered outliers
2. ride_length that falls above 115.63 minutes (approx. 2 hours) are considered outliers

Again, I count the percentages of outliers in the data:

outliers <- which(df_2021_test$ride_length < lower_bound | df_2021_test$ride_length > upper_bound)
(count(df_2021_test[outliers, ]) / count(df_2021_test)) * 100

         n
1 1.982182

The result is that 1.98 % of data are considered outliers . I think this is fine to exclude for the analysis to begin as it will not reduce the accuracy of the analysis that much. Or am I wrong?

Here are my questions:

1. When identifying outliers in the data, what should you choose between the two of the method above? Or is there another better method?
2. Is my way of identifying outliers in the dataset correct? Or am I missing something?

I have detailed all of my steps to identify outliers above and again, it's not an error in the code it's just that I'm confused as to whether my method of identifying them is correct or if is there any better way or something that I miss.

Answer 1

It should be obvious that 50% of your data will fall above the 75th or below the 25th centile, and that 2% is either above the 99th or below the 1st.

There is no data-driven answer to 'what is an outlier'. Data points might be far away from the rest of a distribution for many different reasons, and how you identify these and what you do with them should depend on why you think they occurred and what downstream analysis you are planning (ultimately what question you are asking of the data).

If you can specify these things you'll have a clearer idea of how to identify and handle outliers. Do not rely on simple data-driven rules.

Answer 2

George is correct. Rarely do data-driven "rules" behind outlier detection/removal work alone. As an example using the starwars dataset in R, I have plotted the heights and masses of the Star Wars movie characters below:

#### Load Library ####
library(tidyverse)
theme_set(theme_bw())

#### Plot Obvious Outlier ####
starwars %>% 
  ggplot(aes(x=mass,
             y=height))+
  geom_point()+
  geom_smooth(method = "lm")+
  labs(x="Mass",
       y="Height",
       title = "Star Wars: Mass x Height")

You can see a very obvious outlier. This is Jabba the Hut, whose mass is several leagues above the others in this data. Here it is reasonable to assume that it greatly affects the regression plotted and doesn't model this relationship in a very accurate way.

We can remove it simply by filtering it with this code:

#### Remove Outlier and Plot ####
starwars %>% 
  filter(mass < 1000) %>% 
  ggplot(aes(x=mass,
             y=height))+
  geom_point()+
  geom_smooth(method = "lm")+
  labs(x="Mass",
       y="Height",
       title = "Star Wars: Mass x Height")

Now let's say we didn't have this very obvious outlier and we already started with this particular subset of the data. Then after, we tried to flag the outliers using an often used 1.5xx IQR method to decide what to remove:

#### Highlight Outliers in Plot ####
starwars %>% 
  filter(mass < 1000) %>% 
  mutate(outlier.height = rstatix::is_outlier(height),
         outlier.mass = rstatix::is_outlier(mass)) %>% 
  ggplot(aes(x=mass,
             y=height))+
  geom_point(aes(color=outlier.height))+
  geom_smooth(method = "lm")+
  labs(x="Mass",
       y="Height",
       title = "Star Wars: Mass x Height",
       color = "Height Outlier?")

You will notice that not only are a substantial amount of outliers now present in the data, but shaving them would fundamentally alter the real associations in the data:

Therefore, it is best to decide what is a meaningful inclusion/disclusion of an outlier. In the case of Jabba the Hut, even his separation from the pack has questionable outcomes (there may be others from his species who are also as large, and we may want to predict that in some way). With this being the case, outliers require more thought than a quantile rule.