How to filter a stream of data

问题

Im trying to implement a filter for sensor readings on a Raspi. I get a constant stream of data from the sensors that I want to filter using my filter code. I tried to emulate the sort of readings I would get from a sensor by generating random numbers in my rangen code. The issue is that for some reason I cannot see any output from the filter and I dont even know if it works. Below are the codes for generating a random stream of data and the filter. Am I emulating the sensor readings right?(Note: Im not particularly concerned about the values, I just want a random stream of data that I can keep reading and filter)

Generating random data

# generate random integer values, Filename: Rangen.py
from random import seed
from random import randint
from time import sleep
# seed random number generator
seed(1)
# generate some integers
def rangen():
    value = randint(0, 100)
    return value

while True:
    data = rangen()
    sleep(0.1)

and here is the filter

from time import sleep
from statistics import mean
from Rangen import data

def AplhaTrimmedfilter(windowsize,alpha, sensordata):
    data_list = []
    Flag = True
    while Flag:
        if len(data_list)<windowsize:
            data_list.append(sensordata)
            print("data added")
            sleep(0.0010)
            break
    if len(data_list)==windowsize:
        sorted_data = data_list.sort()     
        sliced_data = sorted_data[(windowsize/2):windowsize-alpha+(alpha/2)]
        alphamean = statistics.mean(sliced_data) #sum(a) / len(a)
        data_list = []
        print(data_list)
    return alphamean

AlphaTrimmedfilter(5,2,data)

I want to take 5 values from the generated data, sort them, slice them and avergae them and display at the end. I dont understand if I implemented this right because the output on the console shows nothing. Any input is appreciated. Thank you.

Edit: As MisterMiyagi suggested, I used generators to tackle the issue with filtering and I iterated over the generator object using [[item] for item in filtered_data] but now I have a weird TypeError 'Float' object is not iterable. How would I fix this issue? I think this might be due to the type of data I pass into the filter(which is mostly float). Here is the code for the modified filter based on MisterMiyagi's answer:

def alpha_mean(window, alpha):
    cut = alpha//2
    data = sorted(window)[cut:-cut]
    result = sum(data)/len(data)
    return result

def alphatrimmer(window_size, alpha, sensor_stream):
    window = []
    for item in sensor_stream:
        window.append(item)
        if len(window) >= window_size:
            break
    yield alpha_mean(window, alpha)

    for item in sensor_stream:
        window.pop(0)
        window.append(item)
        yield alpha_mean(window,alpha)

回答1:

Python's native equivalent of streams are iterators. You can create your own iterator by writing a generator. For example, we can turn your rangen function producing one value into a generator function producing many values.

# stream.py
import random

random.seed(1)

def random_stream():
    """Generator producing a stream of random numbers"""
    while True:      # generators can produce values indefinitely...
        value = random.randint(0, 100)
        yield value  # ... by yield'ing them

Among other things, iterators/generators can be consumed by for statements. You can test this on the Python console:

>>> from stream import random_stream
>>> data = random_stream()   # note: the generator is called using ()
>>> for idx, item in enumerate(data):
...     if idx >= 3: break
...     print(idx, ':', item)
0 : 17
1 : 72
2 : 97

---

A function that consumes such a stream ideally is also a generator - instead of applying a fixed window once, it moves a window over the stream. This allows to seamlessly consume the entire stream, no matter how long it is.

Ideally, you split the tracking of the window and the computation at each window position. Notably, the latter is not a generator - it just works on a single window state.

def alpha_mean(window, alpha):
    """Compute the mean for a given window and alpha"""
    size = len(window)
    data = sorted(window)[size//2:size - (alpha // 2)]
    return sum(data) / len(data)


def alpha_trim(window_size, alpha, sensor_stream):
    """Produce a trimmed stream based on the ``sensor_stream`` data"""
    window = []  # note: a collections.deque is more efficient for very large windows
    # fill the first window
    for item in sensor_stream:
        window.append(item)
        if len(window) >= window_size:
            break
    yield alpha_mean(window, alpha)
    # advance the window as new data comes in
    for item in sensor_stream:
        # "move" the window by discarding the oldest value
        window.pop(0)
        window.append(item)
        yield alpha_mean(window, alpha)

The alpha_trim is a generator that also takes a generator. You can test this again in the Python console:

>>> data = alpha_trim(5, 2, rangen())
>>> for idx, item in enumerate(data):
...     if idx >= 5: break
...     print(idx, ':', item)
0 : 52.0
1 : 52.0
2 : 47.5
3 : 47.5
4 : 60.0

来源：https://stackoverflow.com/questions/59734521/how-to-filter-a-stream-of-data