Kafka topic no longer exists after restart

Question

I created a topic in my local kafka cluster with 3 servers / brokers by running the following from my kafka installation directory

bin/kafka-topics.sh --create         


        

Answer 1

kafka-topics.sh actually uses zookeeper data under the hood to answer the query. The rationale being that a single broker generally can't have enough information by itself to describe topics completely.

If you lost (which I suspect you did, since you mention a new zookeeper start) zookeeper data during your restart process, kafka-topics is now totally blind and can't see former kafka data.

The best way to check what's happening is to actually do what kafka is doing when you query it ! Launch your zookeeper client (it's as simple as doing ./zkCli.sh, and type ls /brokers/topics. If it's empty, your ZK data is lost.

Answer 2

I think you've run into the problem of the /tmp directory being cleaned out whenever your computer reboots. You either need to change the directory that you're storing your Kafka logs to, or change the $TMPTIME environmental variable in /etc/default/rcS which controls how long the tmp files are kept around (time in days).

https://askubuntu.com/questions/20783/how-is-the-tmp-directory-cleaned-up

Answer 3

It doesn't mean if you have the broker directories on temp that you had data on it, brokers tends to create these directories if they doesn't exist.

• Could you try to recreate the topic, restart the machine then have a look on the /tmp directory before starting Kafka ?
• Could you try to reproduce with changing the data directory to something other than /tmp ?