changing kafka retention period during runtime

问题

With Kafka 0.8.1.1, how do I change the log retention time while it's running? The documentation says the property is log.retention.hours, but trying to change it using kafka-topics.sh returns this error

$ bin/kafka-topics.sh --zookeeper zk.yoursite.com --alter --topic as-access --config topic.log.retention.hours=24
Error while executing topic command requirement failed: Unknown configuration "topic.log.retention.hours".
java.lang.IllegalArgumentException: requirement failed: Unknown configuration "topic.log.retention.hours".
    at scala.Predef$.require(Predef.scala:145)
    at kafka.log.LogConfig$$anonfun$validateNames$1.apply(LogConfig.scala:138)
    at kafka.log.LogConfig$$anonfun$validateNames$1.apply(LogConfig.scala:137)
    at scala.collection.Iterator$class.foreach(Iterator.scala:631)
    at scala.collection.JavaConversions$JEnumerationWrapper.foreach(JavaConversions.scala:479)
    at kafka.log.LogConfig$.validateNames(LogConfig.scala:137)
    at kafka.log.LogConfig$.validate(LogConfig.scala:145)
    at kafka.admin.TopicCommand$.parseTopicConfigsToBeAdded(TopicCommand.scala:171)
    at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:95)
    at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:93)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
    at kafka.admin.TopicCommand$.alterTopic(TopicCommand.scala:93)
    at kafka.admin.TopicCommand$.main(TopicCommand.scala:52)
    at kafka.admin.TopicCommand.main(TopicCommand.scala)

回答1:

log.retention.hours is a property of a broker which is used as a default value when a topic is created. When you change configurations of currently running topic using kafka-topics.sh, you should specify a topic-level property.

A topic-level property for log retention time is retention.ms.

From Topic-level configuration in Kafka 0.8.1 documentation:

Property: retention.ms
Default: 7 days
Server Default Property: log.retention.minutes
Description: This configuration controls the maximum time we will retain a log before we will discard old log segments to free up space if we are using the "delete" retention policy. This represents an SLA on how soon consumers must read their data.

So the correct command is

$ bin/kafka-topics.sh --zookeeper zk.yoursite.com --alter --topic as-access --config retention.ms=86400000

You can check whether the configuration is properly applied with the following command.

$ bin/kafka-topics.sh --describe --zookeeper zk.yoursite.com --topic as-access

Then you will see something like below.

Topic:as-access  PartitionCount:3  ReplicationFactor:3  Configs:retention.ms=86400000

回答2:

The following is the right way to alter topic config as of Kafka 0.10.2.0:

bin/kafka-configs.sh --zookeeper <zk_host> --alter --entity-type topics --entity-name test_topic --add-config retention.ms=86400000

Topic config alter operations have been deprecated for bin/kafka-topics.sh.

WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
     Going forward, please use kafka-configs.sh for this functionality`

回答3:

The correct config key is retention.ms

$ bin/kafka-topics.sh --zookeeper zk.prod.yoursite.com --alter --topic as-access --config retention.ms=86400000
Updated config for topic "my-topic".

回答4:

I tested and used this command in kafka confluent V4.0.0 and apache kafka V 1.0.0 and 1.0.1

/opt/kafka/confluent-4.0.0/bin/kafka-configs --zookeeper XX.XX.XX.XX:2181 --entity-type topics --entity-name test --alter --add-config  retention.ms=55000

test is the typic name.

I think it works well in other versions too

来源：https://stackoverflow.com/questions/29129222/changing-kafka-retention-period-during-runtime

标签

apache-kafka

retention