Why does zookeeper not use my log4j.properties file log directory

Question

In my zookeeper/conf/log4j.properties file I set the zookeeper.log.dir to $HOME/zklogs

When I use zkServer.sh it does not use that directory. Instead it uses the ${Z

Answer 1

As far as I can see from zkEnv.sh zookeeper/conf folder is already in the classpath. There are two problems there:

1. zkServer.sh adds "-Dzookeeper.log.dir=." to ZK launch command unless you specify ZOO_LOG_DIR environment variable

2. even if you do specify ZOO_LOG_DIR, ZK may still be unable to find your log4j.properties file, because zkEnv.sh specifies CLASSPATH like this:

   $CLASSPATH="$ZOOCFGDIR:$CLASSPATH"

this works fine if your $CLASSPATH environment variable is not empty, however when it is empty, it just leaves a trailing colon, which screws up your classpath. I changed the lines above to this:

#add the zoocfg dir to classpath
if [ "x${CLASSPATH}" = "x" ]
then
    CLASSPATH="$ZOOCFGDIR"
else
    CLASSPATH="$ZOOCFGDIR:$CLASSPATH"
fi

and also passed ZOO_LOG_DIR environment for ZK, because zkServer.sh generates -Dzookeeper.log.dir VM argument when starting up ZK. Here is the command that starts ZooKeeper for me, makes it read /opt/zookeeper/conf/log4j.properties and keep log files in /opt/zimbra/log:

ZOO_LOG_DIR=/opt/zimbra/log /opt/zookeeper/bin/zkServer.sh start

Answer 2

Another option to change the file log directory via the ZOO_LOG_DIR environment variable is to create your own zk-server-1/conf/zookeeper-env.sh.

Reading the zookeeper-3.4.6 dist bash scripts(t.i. zkEnv.sh), they source that zookeeper-env.sh file if it exists. Here is it's(my created) content:

export ZOO_HOME=$ZOOKEEPER_PREFIX
echo $ZOO_HOME
export ZOO_LOG_DIR=$ZOO_HOME/logs

I am used to having a PRODUCTNAME_HOME variable, which points to the top installation dir of zookeeper. This top dir seems to be named $ZOOKEEPER_PREFIX by zookeeper dist, which i aliase with $ZOO_HOME und use it for the ZOO_LOG_DIR environment variable.

At the end of this post i spooled out the bash scripts debug info - helps me investigate which environment or local variables plus values are used.

As i see it, the /home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf dir is on the CLASSPATH, hence the log4j.properties file is eligible for resource loading via java's class-loader for the log4j framework(which is the way log4j is implemented AFAIR).

joma@kopernikus-u:~/dev/programme/zk-local-ensemble$ bash -x zk-server-1/bin/zkServer.sh start
+ '[' x = x ']'
+ JMXLOCALONLY=false
+ '[' x = x ']'
+ echo 'JMX enabled by default'
JMX enabled by default
+ ZOOMAIN='-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain'
+ ZOOBIN=zk-server-1/bin/zkServer.sh
++ dirname zk-server-1/bin/zkServer.sh
+ ZOOBIN=zk-server-1/bin
++ cd zk-server-1/bin
++ pwd
+ ZOOBINDIR=/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin
+ '[' -e zk-server-1/bin/../libexec/zkEnv.sh ']'
+ . /home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/zkEnv.sh
++ ZOOBINDIR=/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin
++ ZOOKEEPER_PREFIX=/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/..
++ '[' x = x ']'
++ '[' -e /home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf ']'
++ ZOOCFGDIR=/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf
++ '[' -f /home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf/zookeeper-env.sh ']'
++ . /home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf/zookeeper-env.sh
+++ export ZOO_HOME=/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/..
+++ ZOO_HOME=/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/..
+++ echo /home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/..
/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/..
+++ export ZOO_LOG_DIR=/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../logs
+++ ZOO_LOG_DIR=/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../logs
++ '[' x = x ']'
++ ZOOCFG=zoo.cfg
++ ZOOCFG=/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf/zoo.cfg
++ '[' -f /home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf/java.env ']'
++ '[' x/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../logs = x ']'
++ '[' x = x ']'
++ ZOO_LOG4J_PROP=INFO,CONSOLE
++ '[' /home/joma/entwicklung/programme/jdk1.7.0_51 '!=' '' ']'
++ JAVA=/home/joma/entwicklung/programme/jdk1.7.0_51/bin/java
++ CLASSPATH=/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf:
++ for i in '"$ZOOBINDIR"/../src/java/lib/*.jar'
++ CLASSPATH='/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../src/java/lib/*.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf:'
++ '[' -e '/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../share/zookeeper/zookeeper-*.jar' ']'
++ for i in '"$ZOOBINDIR"/../zookeeper-*.jar'
++ CLASSPATH='/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../zookeeper-3.4.6.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../src/java/lib/*.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf:'
++ LIBPATH=("${ZOOBINDIR}"/../lib/*.jar)
++ for i in '"${LIBPATH[@]}"'
++ CLASSPATH='/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/jline-0.9.94.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../zookeeper-3.4.6.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../src/java/lib/*.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf:'
++ for i in '"${LIBPATH[@]}"'
++ CLASSPATH='/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/log4j-1.2.16.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/jline-0.9.94.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../zookeeper-3.4.6.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../src/java/lib/*.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf:'
++ for i in '"${LIBPATH[@]}"'
++ CLASSPATH='/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/netty-3.7.0.Final.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/log4j-1.2.16.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/jline-0.9.94.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../zookeeper-3.4.6.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../src/java/lib/*.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf:'
++ for i in '"${LIBPATH[@]}"'
++ CLASSPATH='/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/slf4j-api-1.6.1.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/netty-3.7.0.Final.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/log4j-1.2.16.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/jline-0.9.94.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../zookeeper-3.4.6.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../src/java/lib/*.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf:'
++ for i in '"${LIBPATH[@]}"'
++ CLASSPATH='/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/slf4j-api-1.6.1.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/netty-3.7.0.Final.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/log4j-1.2.16.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/jline-0.9.94.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../zookeeper-3.4.6.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../src/java/lib/*.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf:'
++ for d in '"$ZOOBINDIR"/../build/lib/*.jar'
++ CLASSPATH='/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../build/lib/*.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/slf4j-api-1.6.1.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/netty-3.7.0.Final.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/log4j-1.2.16.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/jline-0.9.94.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../zookeeper-3.4.6.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../src/java/lib/*.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf:'
++ CLASSPATH='/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../build/classes:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../build/lib/*.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/slf4j-api-1.6.1.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/netty-3.7.0.Final.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/log4j-1.2.16.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/jline-0.9.94.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../zookeeper-3.4.6.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../src/java/lib/*.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf:'
++ case "`uname`" in
+++ uname
++ cygwin=false
++ false
+ '[' x '!=' x ']'
+ '[' x '!=' x ']'
++ dirname /home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf/zoo.cfg
+ '[' x/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf '!=' x/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf ']'
+ false
+ KILL=kill
+ echo 'Using config: /home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf/zoo.cfg'
Using config: /home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf/zoo.cfg
+ '[' -z '' ']'
++ grep '^[[:space:]]*dataDir' /home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf/zoo.cfg
++ sed -e 's/.*=//'
+ ZOO_DATADIR=/home/joma/dev/programme/zk-local-ensemble/data/zk1
+ '[' '!' -d /home/joma/dev/programme/zk-local-ensemble/data/zk1 ']'
+ ZOOPIDFILE=/home/joma/dev/programme/zk-local-ensemble/data/zk1/zookeeper_server.pid
+ '[' '!' -w /home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../logs ']'
+ _ZOO_DAEMON_OUT=/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../logs/zookeeper.out
+ case $1 in
+ echo -n 'Starting zookeeper ... '
Starting zookeeper ... + '[' -f /home/joma/dev/programme/zk-local-ensemble/data/zk1/zookeeper_server.pid ']'
+ '[' 0 -eq 0 ']'
+ /bin/echo -n 24744
+ nohup /home/joma/entwicklung/programme/jdk1.7.0_51/bin/java -Dzookeeper.log.dir=/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../logs -Dzookeeper.root.logger=INFO,CONSOLE -cp '/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../build/classes:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../build/lib/*.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/slf4j-api-1.6.1.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/netty-3.7.0.Final.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/log4j-1.2.16.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../lib/jline-0.9.94.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../zookeeper-3.4.6.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../src/java/lib/*.jar:/home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf:' -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /home/joma/dev/programme/zk-local-ensemble/zk-server-1/bin/../conf/zoo.cfg
+ sleep 1
+ echo STARTED
STARTED

Answer 3

zkServer.sh gets it's environment variables from zkEnv.sh The env file sets a classpath which includes the log4j file if it's at the expected location.

ZOOXFGDIR=ZOOBINDIR/../conf

I dropped some echos into zkServer.sh to trace what's going on. I found that classpath was being set properly, but logdir and log4j_prop were not being set. So I added them to zkEnv.sh. Logs appear to be showing up in the expected location now.

ZOO_LOG_DIR="/var/log/zookeeper"
ZOO_LOG4J_PROP="INFO,ROLLINGFILE"

Answer 4

I wanted to add how I fixed this problem / customized my environment.

There are 2 logging mechanisms working here:

• bin/zkServer.sh redirects the zookeeper server's stdout and stderr to zookeeper.out
• log4j can append to logs to several places including:
  • CONSOLE - which ends up in zookeeper server's stdout and stderr
  • ROLLINGFILE - which is sent to zookeeper.log

bin/zkServer.sh uses :

• ZOO_LOG_DIR to set the path for both zookeeper.out and log4j.
• ZOO_LOG4J_PROP to set the log4j logging level and what log appenders are turned on

The "eventual" defaults in the setup conf/log4j.properties are set by a combination of zookeeper bash scripts:

• ZOO_LOG_DIR = . ( the working directory from which zookeeper is started )
  • set inside of conf/log4j.properties as zookeeper.log.dir
• ZOO_LOG4J_PROP = INFO, CONSOLE
  • set inside of conf/log4j.properties as zookeeper.root.logger

The effect of turning on the log appender CONSOLE is that logs now go to stdout. Because bin/zkServer.sh redirects stdout and stderr to zookeeper.out, log4j logs end up in zookeeper.out. The effect of turning off ROLLINGFILE is that no zookeeper.log file is created.

The zookeeper.out log is not rotated. The zookeeper.log log is set to be rotated and can be set to expire old logs.

I wanted logs to roll and be expired. The first thing I had to do was change conf/log4j.properties to cause the expiration/deletion of old logs. I did that by setting log4j.appender.ROLLINGFILE.MaxBackupIndex inside of conf/log4j.properties. The second thing I had to do was set the log directory, logging level and appenders.

I have a bash script that runs every minute. If it sees that zookeeper isn't running, it runs :

bin/zkServer.sh start

I changed it to specify environmental variables expected by bin/zkServer.sh.

sudo ZOO_LOG_DIR=/opt/zookeeper-3.4.6/logs ZOO_LOG4J_PROP='INFO,ROLLINGFILE' /opt/zookeeper-3.4.6/bin/zkServer.sh start

The effect of turning off the log appender CONSOLE is that log4j logs now no longer end up in zookeeper.out. The effect of turning on ROLLINGFILE is that zookeeper.log file is created, rotated, and expired.

BTW, conf/log4j.properties was apparently already in my classpath. I had to make no changes in that regard.

This chain contributed significantly to my understanding: https://groups.google.com/forum/#!msg/nosql-databases/aebIvNnT0xY/doky1X9-WfwJ

Answer 5

Maybe not the sought answer, but for zookeeper-3.4.11 in the conf directory you can set the appropriate levels and configuration in the file log4j.properties

Answer 6

I believe the answer to my question is that I need to move that log4j.properties file into my classpath and then and only then will zkEnv.sh read of the values that I set in log4j.properties when I call zkServer.sh. I believe I just did not understand how to use log4j properly.