Kafka SessionWindow with suppress only sends final event when there is a steady stream of input records

问题

It appears as though a Kafka Stream with a session window with a grace period and suppression fails to output a final event if there is no constant stream of input records.

Context: We are using change data capture (CDC) to monitor changes to a legacy database. When a user makes changes using the UI, a database transaction will change 1..n tables. Each SQL statement results in a Kafka record. These need to be aggregated in order to create one "trigger record" which is used to start an expensive process. The process should be started within a second of the transaction in the legacy database being committed. There are only a handful of users working with the old application, and so there can be significant amount of time between transactions.

We have a Kafka Stream application which uses a session window and an inactivity gap of 400ms in order to aggregate the incoming records that share the same key (the transaction ID), and output the trigger record.

We have a working solution but the trigger record is only written to the output topic so long as other transactions are running in order to generate a steady stream of incoming records. We need the window to close and the trigger record to be written, even if there are no further input records.

Working code is here: https://github.com/maxant/kafka-data-consistency/blob/714a44689fd48aa28b05b855638ac7ed50dd5ee9/partners/src/main/java/ch/maxant/kdc/partners/ThroughputTestStream.java#L65

Here is a summary of that code:

      stream.groupByKey()
            .windowedBy(SessionWindows.with(Duration.ofMillis(400)).grace(Duration.ofMillis(0)))
            .aggregate(...)
            .suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded()))
            .toStream((k,v) -> k.key())
            .to("throughput-test-aggregated");

Initially I had no suppression and no grace period. Using just the default configuration, I always received a final event for the window containing all the aggregated records, but it was taking up to 6 seconds after the 400ms window, which is too long for us to wait.

In order to reduce the latency and speed things up, I set CACHE_MAX_BYTES_BUFFERING_CONFIG to 1, but that caused an output record after each aggregation, rather than just a single output record.

I introduced the suppression (and with it, a grace period of 0ms), in order to ensure that only one output record is created.

The problem now is that I only receive an output record, if new input records arrive after the window closes (regardless of their key).

The test creates 10 input records all with the same key, 10ms apart, all within 100ms. It then rests for 3 seconds, allowing me to shut it off after one group of ten records. I expect to receive one output record, but none arrives, unless I leave the test running, to create a second group of input records. This problem is reproducible.

I have read the following articles but cannot find anything which describes what I am seeing, namely that the final record is only sent to the output topic once additional records (regardless of key) are processed.

https://www.confluent.io/blog/kafka-streams-take-on-watermarks-and-triggers/
https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables
https://kafka.apache.org/10/documentation/streams/developer-guide/config-streams.html

What do I have to change so that the final record is sent to my output topic, even if no further records are processed?

(Using Kafka 2.4.1 with client and server on Linux)

回答1:

Update : I have an error in topology, fixed

I had the same exactly same problems as you when using suppress, and it's expected behavior. Because suppress only support emit buffered records using stream time not wall-clock time, if you stop getting new records, stream time will be freeze and Suppress will not emit the last suppressed window.

The solution I used is to write a custom suppress using Processor API (use a Transfomer so you can use DSL to send supprssed record downstream) with a state store used as a buffer, then check what windows should be flush (or emit) to downstream processor whenever there is a new record come in or after a time interval has passed (using a WALL_CLOCK_TIME punctuate).

The transfomer would look like this:

public class SuppressWindowTransformer implements Transformer<Windowed<String>, String, KeyValue<Windowed<String>, String>> {
    private ProcessorContext context;
    private Cancellable cancellable;
    private KeyValueStore<Windowed<String>, String> kvStore;
    @Override
    public void init(ProcessorContext context) {
        this.context = context;
        kvStore = (KeyValueStore) context.getStateStore("suppressed_windowed_transaction");
        cancellable = context.schedule(Duration.ofMillis(100), PunctuationType.WALL_CLOCK_TIME, timestamp -> flushOldWindow());
    }

    @Override
    public KeyValue<Windowed<String>, String> transform(Windowed<String> key, String value) {
        kvStore.put(key, value);//buffer (or suppress) the new in coming window
        flushOldWindow();
        return null;
    }

    private void flushOldWindow() {
        //your logic to check for old windows in kvStore then flush

        //forward (or unsuppressed) your suppressed records downstream using ProcessorContext.forward(key, value)
    }

    @Override
    public void close() {
        cancellable.cancel();//cancel punctuate
    }
}

And in your Stream DSL:

stream.groupByKey()
            .windowedBy(SessionWindows.with(Duration.ofMillis(400)).grace(Duration.ofMillis(0)))
            .aggregate(...)//remove suppress operator and write custom suppress using processor API
            .toStream()
            .transform(SuppressWindowTransformer::new, "suppressed_windowed_transaction")
            .to("throughput-test-aggregated");

来源：https://stackoverflow.com/questions/60822669/kafka-sessionwindow-with-suppress-only-sends-final-event-when-there-is-a-steady

标签

apache-kafka-streams