Matlab - Signal Noise Removal

Question

I have a vector of data, which contains integers in the range -20 20.

Bellow is a plot with the values:

Answer 1

What you actually need is some kind of compression to scale your data, that is: values between -2 and 2 are scale by a certain factor and everything else is scaled by another factor. A crude way to accomplish such a thing, is by putting all small values to zero, i.e.

x = randn(1,100)/2; x(50) = 20; x(25) = -15; % just generating some data
threshold = 2;
smallValues = (abs(x) <= threshold);
y = x;
y(smallValues) = 0;
figure; 
plot(x,'DisplayName','x'); hold on; 
plot(y,'r','DisplayName','y'); 
legend show;

Please do not that this is a very nonlinear operation (e.g. when you have wanted peaks valued at 2.1 and 1.9, they will produce very different behavior: one will be removed, the other will be kept). So for displaying, this might be all you need, for further processing it might depend on what you are trying to do.

Answer 2

You might try a split window filter. If x is your current sample, the filter would look something like:

k = [L L L L L L 0 0 0 x 0 0 0 R R R R R R]

For each sample x, you average a band of surrounding samples on the left (L) and a band of surrounding samples on the right. If your samples are positive and negative (as yours are) you should take the abs. value first. You then divide the sample x by the average value of these surrounding samples.

y[n] = x[n] / mean(abs(x([L R])))

Each time you do this the peaks are accentuated and the noise is flattened. You can do more than one pass to increase the effect. It is somewhat sensitive to the selection of the widths of these bands, but can work. For example:

Two passes:

Answer 3

If it's for demonstrative purposes only, and you're not actually going to be using these scaled values for anything, I sometimes like to increase contrast in the following way:

% your data is in variable 'a'
plot(a.*abs(a)/max(abs(a)))

edit: since we're posting images, here's mine (before/after):

Answer 4

One approach to detect outliers is to use the three standard deviation rule. An example:

%# some random data resembling yours
x = randn(100,1);
x(75) = -14;
subplot(211), plot(x)

%# tone down the noisy points
mu = mean(x); sd = std(x); Z = 3;
idx = ( abs(x-mu) > Z*sd );         %# outliers
x(idx) = Z*sd .* sign(x(idx));      %# cap values at 3*STD(X)
subplot(212), plot(x)

---

EDIT:

It seems I misunderstood the goal here. If you want to do the opposite, maybe something like this instead:

%# some random data resembling yours
x = randn(100,1);
x(75) = -14; x(25) = 20;
subplot(211), plot(x)

%# zero out everything but the high peaks
mu = mean(x); sd = std(x); Z = 3;
x( abs(x-mu) < Z*sd ) = 0;
subplot(212), plot(x)

Answer 5

To eliminate the low amplitude peaks, you're going to equate all the low amplitude signal to noise and ignore.

If you have any apriori knowledge, just use it.

if your signal is a, then

a(abs(a)<X) = 0

where X is the max expected size of your noise.

If you want to get fancy, and find this "on the fly" then, use kmeans of 3. It's in the statistics toolbox, here:

http://www.mathworks.com/help/toolbox/stats/kmeans.html

Alternatively, you can use Otsu's method on the absolute values of the data, and use the sign back.

Note, these and every other technique I've seen on this thread is assuming you are doing post processing. If you are doing this processing in real time, things will have to change.