Python: Find outliers inside a list

问题

I'm having a list with a random amount of integers and/or floats. What I'm trying to achieve is to find the exceptions inside my numbers (hoping to use the right words to explain this). For example:

list = [1, 3, 2, 14, 108, 2, 1, 8, 97, 1, 4, 3, 5]

• 90 to 99% of my integer values are between 1 and 20
• sometimes there are values that are much higher, let's say somewhere around 100 or 1.000 or even more

My problem is, that these values can be different all the time. Maybe the regular range is somewhere between 1.000 to 1.200 and the exceptions are in the range of half a million.

Is there a function to filter out these special numbers?

回答1:

Assuming your list is l:

• If you know you want to filter a certain percentile/quantile, you can use:

  This removes bottom 10% and top 90%. Of course, you can change any of them to your desired cut-off (for example you can remove the bottom filter and only filter the top 90% in your example):

  import numpy as np
  l = np.array(l)
  l = l[(l>np.quantile(l,0.1)) & (l<np.quantile(l,0.9))].tolist()
  

  output:

  [ 3  2 14  2  8  4  3  5]
  

• If you are not sure of the percentile cut-off and are looking to remove outliers:

  You can adjust your cut-off for outliers by adjusting argument m in function call. The larger it is, the less outliers are removed. This function seems to be more robust to various types of outliers compared to other outlier removal techniques.

   import numpy as np 
   l = np.array(l) 
   def reject_outliers(data, m=6.):
      d = np.abs(data - np.median(data))
      mdev = np.median(d)
      s = d / (mdev if mdev else 1.)
      return data[s < m].tolist()
   print(reject_outliers(l))
  

  output:

  [1, 3, 2, 14, 2, 1, 8, 1, 4, 3, 5]
  

回答2:

You can use the built-in filter() method:

lst1 = [1, 3, 2, 14, 108, 2, 1, 8, 97, 1, 4, 3, 5]

lst2 = list(filter(lambda x: x > 5,lst1))

print(lst2)

Output:

[14, 108, 8, 97]

回答3:

So here is a method how to block out those deviators

import math
_list = [1, 3, 2, 14, 108, 2, 1, 8, 97, 1, 4, 3, 5]
def consts(_list):
    mu = 0
    for i in _list:
        mu += i
    mu = mu/len(_list)
    sigma = 0
    for i in _list:
        sigma += math.pow(i-mu,2)
    sigma = math.sqrt(sigma/len(_list))
    return sigma, mu

def frequence(x, sigma, mu):
    return (1/(sigma*math.sqrt(2*math.pi)))*math.exp(-(1/2)*math.pow(((x-mu)/sigma),2))

sigma, mu = consts(_list)

new_list = []
for i in range(len(_list)):
    if frequence(_list[i], sigma, mu) > 0.01:
        new_list.append(i)
print(new_list)

来源：https://stackoverflow.com/questions/62802061/python-find-outliers-inside-a-list