Finding and removing outliers in PHP

Question

Suppose I sample a selection of database records that return the following numbers:

20.50, 80.30, 70.95, 15.25, 99.97, 85.56, 69.77

Is ther

Answer 1

For a normally distributed set of data, removes values more than 3 standard deviations from the mean.

<?php
function remove_outliers($array) {
    if(count($array) == 0) {
      return $array;
    }
    $ret = array();
    $mean = array_sum($array)/count($array);
    $stddev = stats_standard_deviation($array);
    $outlier = 3 * $stddev;
    foreach($array as $a) {
        if(!abs($a - $mean) > $outlier) {
            $ret[] = $a;
        }
    }
    return $ret;
}

Answer 2

Ok let's assume you have your data points in an array like so:

<?php $dataset = array(20.50, 80.30, 70.95, 15.25, 99.97, 85.56, 69.77); ?>

Then you can use the following function (see comments for what is happening) to remove all numbers that fall outside of the mean +/- the standard deviation times a magnitude you set (defaults to 1):

<?php

function remove_outliers($dataset, $magnitude = 1) {

  $count = count($dataset);
  $mean = array_sum($dataset) / $count; // Calculate the mean
  $deviation = sqrt(array_sum(array_map("sd_square", $dataset, array_fill(0, $count, $mean))) / $count) * $magnitude; // Calculate standard deviation and times by magnitude

  return array_filter($dataset, function($x) use ($mean, $deviation) { return ($x <= $mean + $deviation && $x >= $mean - $deviation); }); // Return filtered array of values that lie within $mean +- $deviation.
}

function sd_square($x, $mean) {
  return pow($x - $mean, 2);
} 

?>

For your example this function returns the following with a magnitude of 1:

Array
(
    [1] => 80.3
    [2] => 70.95
    [5] => 85.56
    [6] => 69.77
)

Answer 3

Topic: Detecting local, additive outliers in unordered arrays by walking a small window through the array and calculating the standard deviation for a certain range of values.

Good morning folks,

here is my solution much to late, but since I was looking for detecting outliers via PHP and could'nt find anything basic, I decided somehow smoothing a given dataset in a timeline of 24 h by simply moving a range of 5 items in a row through an unordered array and calculate the local standard deviation to detect the additive outliers.

The first function will simply calculate the average and deviation of a given array, where $col means the column with the values (sorry for the freegrades, this means that in an uncomplete dataset of 5 values you only have 4 freegrades - I don't know the exact english word for Freiheitsgrade):

function analytics_stat ($arr,$col,$freegrades = 0) {

// calculate average called mu
$mu = 0;
foreach ($arr as $row) {
    $mu += $row[$col];
}
$mu = $mu / count($arr);

// calculate empiric standard deviation called sigma
$sigma = 0;
foreach ($arr as $row) {
    $sigma += pow(($mu - $row[$col]),2);
}
$sigma = sqrt($sigma / (count($arr) - $freegrades));

return [$mu,$sigma];
}

Now its time for the core function, which will move through the given array and create a new array with the result. Margin means the factor to multiply the deviation with, since only one Sigma detects to many outliers, whereas more than 1.7 seems to high:

function analytics_detect_local_outliers ($arr,$col,$range,$margin = 1.0) {

$count = count($arr);
if ($count < $range) return false;

// the initial state of each value is NOT OUTLIER
$arr_result = [];
for ($i = 0;$i < $count;$i++) {
    $arr_result[$i] = false;
}

$max = $count - $range + 1;
for ($i = 0;$i < $max;$i++) {

    // calculate mu and sigma for current interval
    // remember that 5 values will determine the divisor 4 for sigma
    // since we only look at a part of the hole data set
    $stat = analytics_stat(array_slice($arr,$i,$range),$col,1);

    // a value in this interval counts, if it's found outside our defined sigma interval
    $range_max = $i + $range;
    for ($j = $i;$j < $range_max;$j++) {
        if (abs($arr[$j][$col] - $stat[0]) > $margin * $stat[1]) {
            $arr_result[$j] = true;

            // this would  be the place to add a counter to isolate
            // real outliers from sudden steps in our data set
        }
    }
}

return $arr_result;
}

And finally comes the test function with random values in an array with length 24. As for margin I was curious and choose the Golden Cut PHI = 1.618 ... since I really like this number and some Excel test results have led me to a margin of 1.7, above which outliers very rarelly were detected. The range of 5 is variable, but for me this was enough. So for every 5 values in a row there will be a calculation:

function test_outliers () {

// create 2 dimensional data array with items [hour,value]
$arr = [];
for ($i = 0;$i < 24;$i++) {
    $arr[$i] = [$i,rand(0,500)];
}

// set parameter for detection algorithm
$result = [];
$col = 1;
$range = 5;
$margin = 1.618;
$result = analytics_detect_local_outliers ($arr,$col,$range,$margin);

// display results
echo "<p style='font-size:8pt;'>";
for ($i = 0;$i < 24;$i++) {
    if ($result[$i]) echo "&diams;".$arr[$i][1]."&diams; "; else echo $arr[$i][1]." ";
}
echo "</p>";
}

After 20 calls of the test function I got these results:

417 140 372 131 449 26 192 222 320 349 94 147 201 ♦342♦ 123 16 15 ♦490♦ 78 190 ♦434♦ 27 3 276

379 440 198 135 22 461 208 376 286 ♦73♦ 331 358 341 14 112 190 110 266 350 232 265 ♦63♦ 90 94

228 ♦392♦ 130 134 170 ♦485♦ 17 463 13 326 47 439 430 151 268 172 342 445 477 ♦21♦ 421 440 219 95

88 121 292 255 ♦16♦ 223 244 109 127 231 370 16 93 379 218 87 ♦335♦ 150 84 181 25 280 15 406

85 252 310 122 188 302 ♦13♦ 439 254 414 423 216 456 321 85 61 215 7 297 337 204 210 106 149

345 411 308 360 308 346 ♦451♦ ♦77♦ 16 498 331 160 142 102 ♦496♦ 220 107 143 ♦241♦ 113 82 355 114 452

490 222 412 94 2 ♦480♦ 181 149 41 110 220 ♦477♦ 278 349 73 186 135 181 ♦39♦ 136 284 340 165 438

147 311 246 449 396 328 330 280 453 374 214 289 489 185 445 86 426 246 319 ♦30♦ 436 290 384 232

442 302 ♦436♦ 50 114 15 21 93 ♦376♦ 416 439 ♦222♦ 398 237 234 44 102 464 204 421 161 330 396 461

498 320 105 22 281 168 381 216 435 360 19 ♦402♦ 131 128 66 187 291 459 319 433 86 84 325 247

440 491 381 491 ♦22♦ 412 33 273 256 331 79 452 314 485 66 138 116 356 290 190 336 178 298 218

394 439 387 ♦80♦ 463 369 ♦104♦ 388 465 455 ♦246♦ 499 70 431 360 ♦22♦ 203 280 241 319 ♦34♦ 238 439 497

485 289 249 ♦416♦ 228 166 217 186 184 ♦356♦ 142 166 26 91 70 ♦466♦ 177 357 298 443 307 387 373 209

338 166 90 122 442 429 499 293 ♦41♦ 159 395 79 307 91 325 91 162 211 85 189 278 251 224 481

77 196 37 326 230 281 ♦73♦ 334 159 490 127 365 37 57 246 26 285 468 228 181 74 ♦455♦ 119 435

328 3 216 149 217 348 65 433 164 473 465 145 341 112 462 396 168 251 351 43 320 123 181 198

216 213 249 219 ♦29♦ 255 100 216 181 233 33 47 344 383 ♦94♦ 323 440 187 79 403 139 382 37 395

366 450 263 160 290 ♦126♦ 304 307 335 396 458 195 171 493 270 434 222 401 38 383 158 355 311 150

402 339 382 97 125 88 300 332 250 ♦86♦ 362 214 448 67 114 ♦354♦ 140 16 ♦354♦ 109 0 168 127 89

450 5 232 155 159 264 214 ♦416♦ 51 429 372 230 298 232 251 207 ♦322♦ 160 148 206 293 446 111 338

I hope, this will help anyone in the present or future. Greetings

P.S. To further improve this algorithm you may add a counter, which makes sure, that a certain value must for instance be found at least 2 times, that means in 2 different intervals or windows, before it is labeled as outlier. So a sudden jump of the following values does not make the first value the villain. Let me give you an example:

In 3,6,5,9,37,40,42,51,98,39,33,45 there is an obvious step from 9 to 37 and an isolated value 98. I would like to detect 98, but not 9 or 37. The first interval 3,6,5,9,37 would detect 37, the second interval 6,5,9,37,40 not. So we would not detect 37, since there is only one problematic interval or one match. Now it should be clear, that 98 counts in 5 intervals and is therefore an outlier. So lets declare a value an outlier, if it "counts" at least 2 times. Like so often we have to look closely the borders, since they have only one interval, and make for these values an exception.