Delete outliers automatically of a calculated agglomerative hierarchical clustering data

问题

in the cluster analysis the outliers of a dataset can be easily identified by the single-linkage method. Now I would like to remove the outliers automatically. My idea is to remove the data which exceed a specified distance value. Here is my code with the example data of mtcars:

library(cluster)
library(dendextend)
cluster<-agnes(mtcars,stand=FALSE,method="single")
dend = as.dendrogram(cluster)

In the Plot you can see the resulting dendrogram. The last 4 cars ("Duster 360", "Camaro Z28", "Ford Pantera L", "Maserati Bora") are identified outliers so I would like to remove their hole rows(of the dataset mtcars). How can I do it automatically? E.g. remove the rows which height is above 70? I've tried a lot of possibilities to remove outliers but they did not seem to be applicable to my data.

Thanks a lot!

回答1:

try this:

# your code
library(cluster)
cluster<-agnes(mtcars,stand=FALSE,method="single")
dend = as.dendrogram(cluster)
plot(dend)

#new code    
hclu <- as.hclust(cluster) # convert to list that cutree() understands 
groupindexes <- cutree(hclu, h = 70) # cut at height 70 - creates 3 groups/branches
mtcars[groupindexes != 1,] # "outliers" - not in group 1 but in groups 2 and 3
mtcars[groupindexes == 1,] # all but the 4 "outliers"

Result 1 - the "outliers":

                mpg cyl disp  hp drat   wt  qsec vs am gear carb
Duster 360     14.3   8  360 245 3.21 3.57 15.84  0  0    3    4
Camaro Z28     13.3   8  350 245 3.73 3.84 15.41  0  0    3    4
Ford Pantera L 15.8   8  351 264 4.22 3.17 14.50  0  1    5    4
Maserati Bora  15.0   8  301 335 3.54 3.57 14.60  0  1    5    8

Result 2:

                     mpg cyl  disp  hp drat    wt  qsec vs am gear carb
Mazda RX4           21.0   6 160.0 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag       21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4
Datsun 710          22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1
(....and ~30 other rows ....)

回答2:

If your "rule" is the linking distance, then you essentially recreated nearest-neighbor outlier detection, one of the older outlier methods in data-mining.

Ramaswamy, Sridhar, Rajeev Rastogi, and Kyuseok Shim. "Efficient algorithms for mining outliers from large data sets." ACM Sigmod Record. Vol. 29. No. 2. ACM, 2000.

Except that single-link with AGNES takes O(n³) time, but an index can do kNN outlier in O(n log n).

来源：https://stackoverflow.com/questions/46325350/delete-outliers-automatically-of-a-calculated-agglomerative-hierarchical-cluster

标签

cluster-analysis

outliers