Jenkins log getting huge and filling up entire disk space

问题

Every week I got surprised by my Jenkins server hitting 100% disk used by Jenkins log.

So I remove the file, and then my disk gets lots of free space again.

[ec2-user@ip-xxx-xxx-xxx-xxx ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        987M   60K  987M   1% /dev
tmpfs           997M     0  997M   0% /dev/shm
/dev/xvda1       32G   32G     0 100% /         <============================= DISK FULL

[ec2-user@ip-xxx-xxx-xxx-xxx ~]$ sudo rm /var/log/jenkins/jenkins.log <======= REMOVE LOG FILE

[ec2-user@ip-xxx-xxx-xxx-xxx ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        987M   60K  987M   1% /dev
tmpfs           997M     0  997M   0% /dev/shm
/dev/xvda1       32G   11G   21G  35% /         <============================= FREE SPACE AGAIN

My Log Level configs are:

Name                | Level
----------------------------------
winstone            | 5
org.apache.sshd     | WARNING
                    | INFO

How can I definitively prevent this problem?

回答1:

This is a common problem for anyone running from the war, and even originally some of the packaged installers, see JENKINS-8641

By default, Jenkins did not include system log rotation, so eventually it will fill the disk. Eventually determined by how much you use Jenkins and the disk space available.

This cloudbees doc describes how to use logrotate. Link to man page. You can enable even for standalone war execution.

/var/log/jenkins/jenkins.log {
        weekly
        copytruncate
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
}

Normally this file would be found in /etc/logrotate.d/jenkins or /etc/logrotate.d/jenkins-oc.

You may want to set to rotate daily and retain only a week (rotate 7), or set specific limits based on maxsize or a combination.

You can read more about logging in Jenkins, here:

JENKINS - Logging

JENKINS - Logging Configuration