Kafka, new storage

断了今生、忘了曾经 提交于 2020-01-16 06:46:10

问题


I'm trying to add new storage for Kafka, here is what I have already done:

  1. Add, prepare and mount storage under Linux OS
  2. Add new storage in Kafka Broker: log.dirs: /data0/kafka-logs,/data1/kafka-logs
  3. Restart Kafka Brokers
  4. New directories under /data1/kafka-logs has been created but the size is:

du -csh /data1/kafka-logs/ 156K /data1/kafka-logs/

And the size isn't growing only the old /data0 is used.

What I'm missing? What should I do more to solve this problem? The storage is almost full, and the /data1 is still not used.


回答1:


When you add a new data storage path for Kafka, the old data does not start automatically rebalancing on this disc. The new storage will be used when you are, for example, create a new topic, or increase the number of partitions in the existing ones.

In that case, if you need to move some data to a new disk, you will need to use manual data rebalancing. For more details see: https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools

In your case, to prevent the complete exhaustion of disk space, you can for some time to reduce the lifetime of the topic (retention.ms), otherwise the broker will fall.




回答2:


I can only guess - but for me it sounds like you did NOT created a new topic or increased the number of partitions of your current topic(s). Kafka will not move the existing partitions into the new log.dir. According to the kafka documentation:

... Each new partition that is created will be placed in the directory which currently has the fewest partitions. - https://kafka.apache.org/08/configuration.html



来源:https://stackoverflow.com/questions/37735095/kafka-new-storage

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!