Make Kafka Topic Log Retention Permanent

Question

I am writing log messages into a Kafka Topic and I want the retention of this topic to be permanent. I have seen in Kafka and Kafka Connect (_schemas, connect-configs, conne

Answer 1

You can find the default values for parameters here: https://jaceklaskowski.gitbooks.io/apache-kafka/content/kafka-properties.html

log.retention.bytes and log.retention.ms can be set to -1 if you want to keep the data in topics forever.

Answer 2

These special topics are compacted topics. This means they are made up of keyed messages and only keep the list recent key. A full write here. This is probably what you want for infinite retention time in most cases.

Answer 3

If you want to retain all topics forever, you can set both log.retention.hours and log.retention.bytes to -1.