nginx日志切割(logrotate或shell脚本)

大憨熊 提交于 2020-04-28 12:21:30

nginx自己不会对日志文件进行切割,可以通过两种不同的方式进行,分别是:通过logrotate和通过shell脚本。

如果是yum方式安装的nginx,系统默认会自动通过logrotate这个日志管理软件,按天进行分割(配置如下,具体含义可看下文)。

[root@flume-2 ~]# cat /etc/logrotate.d/nginx 
/var/log/nginx/*log {
    create 0644 nginx nginx
    daily
    rotate 10
    missingok
    notifempty
    compress
    sharedscripts
    postrotate
        /bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
    endscript
}

下面分别对这两种方法进行介绍下:

一、使用logrotate切割

前面说了yum或者apt-get方式安装的nginx会自动使用logrotate这个日志管理软件进行切割,所以本章节我们主要介绍有关logrotate相关的知识点。

1.1 logrotate介绍

logrotate是什么呢?它是一个linux系统日志的管理工具。它可以切割、压缩等其他软件的日志文件软件。

logrotate是基于crontab运行的,所以这个时间点是由crontab控制的,具体可以查询crontab的配置文件/etc/anacrontab。

我们可以通过如下命令安装logrotate,如下:

yum -y install logrotate

如果是ubuntu系统,可以使用如下命令进行安装:

apt-get -y install logrotate

查看logrotate的配置文件,使用如下命令:

[root@flume-2 ~]# rpm -ql logrotate
/etc/cron.daily/logrotate
/etc/logrotate.conf
/etc/logrotate.d
/usr/sbin/logrotate
/usr/share/doc/logrotate-3.8.6
/usr/share/doc/logrotate-3.8.6/CHANGES
/usr/share/doc/logrotate-3.8.6/COPYING
/usr/share/man/man5/logrotate.conf.5.gz
/usr/share/man/man8/logrotate.8.gz
/var/lib/logrotate.status

我们可以很明显的看到logrotate的配置文件是/etc/logrotate.conf,而/etc/logrotate.d/是用于存储其他配置文件的目录。该目录里的所有文件都会被主动的读入 /etc/logrotate.conf中执行。

我使用的配置,用于切割nginx日志:

/usr/local/nginx/logs/*.log {
    daily
    dateext
    rotate 10
    copytruncate
    nocompress
    missingok
    notifempty
    sharedscripts
    postrotate
      /bin/kill -HUP `/bin/cat  /usr/local/nginx/logs/nginx.pid`
    endscript
}

1.2 logrotate配置文件详解

logrotate配置选项,可查看man,如下:

[root@flume-2 ~]# man logrotate
LOGROTATE(8)                                                                      System Administrator's Manual                                                                     LOGROTATE(8)

NAME
       logrotate ‐ rotates, compresses, and mails system logs

SYNOPSIS
       logrotate [-dv] [-f|--force] [-s|--state file] config_file ..

DESCRIPTION
       logrotate  is  designed  to ease administration of systems that generate large numbers of log files.  It allows automatic rotation, compression, removal, and mailing of log files.  Each
       log file may be handled daily, weekly, monthly, or when it grows too large.

       Normally, logrotate is run as a daily cron job.  It will not modify a log multiple times in one day unless the criterion for that log is based on the log's size and logrotate  is  being
       run multiple times each day, or unless the -f or --force option is used.

       Any  number  of  config  files may be given on the command line. Later config files may override the options given in earlier files, so the order in which the logrotate config files are
       listed is important.  Normally, a single config file which includes any other config files which are needed should be used.  See below for more information on how  to  use  the  include
       directive to accomplish this.  If a directory is given on the command line, every file in that directory is used as a config file.

       If  no  command  line  arguments are given, logrotate will print version and copyright information, along with a short usage summary.  If any errors occur while rotating logs, logrotate
       will exit with non-zero status.

OPTIONS
       -?, --help
              Prints help message.

       -d, --debug
              Turns on debug mode and implies -v.  In debug mode, no changes will be made to the logs or to the logrotate state file.

       -f, --force
              Tells logrotate to force the rotation, even if it doesn't think this is necessary.  Sometimes this is useful after adding new entries to a logrotate config file, or  if  old  log
              files have been removed by hand, as the new files will be created, and logging will continue correctly.

       -m, --mail <command>
              Tells  logrotate which command to use when mailing logs. This command should accept two arguments: 1) the subject of the message, and 2) the recipient. The command must then read
              a message on standard input and mail it to the recipient. The default mail command is /bin/mail -s.

       -s, --state <statefile>
              Tells logrotate to use an alternate state file.  This is useful if logrotate is being run as a different user  for  various  sets  of  log  files.   The  default  state  file  is
              /var/lib/logrotate.status.

       --usage
              Prints a short usage message.

       +-v, --verbose
              Turns on verbose mode.

CONFIGURATION FILE
       logrotate  reads  everything  about  the  log  files  it should be handling from the series of configuration files specified on the command line.  Each configuration file can set global
       options (local definitions override global ones, and later definitions override earlier ones) and specify logfiles to rotate. A simple configuration file looks like this:

       # sample logrotate configuration file
       compress

       /var/log/messages {
           rotate 5
           weekly
           postrotate
               /usr/bin/killall -HUP syslogd
           endscript
       }

       "/var/log/httpd/access.log" /var/log/httpd/error.log {
           rotate 5
           mail www@my.org
           size 100k
           sharedscripts
           postrotate
               /usr/bin/killall -HUP httpd
           endscript
       }

       /var/log/news/* {
           monthly
           rotate 2
           olddir /var/log/news/old
           missingok
           postrotate
               kill -HUP `cat /var/run/inn.pid`
           endscript
           nocompress
       }

       ~/log/*.log {}

       The first few lines set global options; in the example, logs are compressed after they are rotated.  Note that comments may appear anywhere in the config file as long as the first  non-
       whitespace character on the line is a #.

       The  next  section  of  the config files defined how to handle the log file /var/log/messages. The log will go through five weekly rotations before being removed. After the log file has
       been rotated (but before the old version of the log has been compressed), the command /sbin/killall -HUP syslogd will be executed.

       The next section defines the parameters for both /var/log/httpd/access.log and /var/log/httpd/error.log.  They are rotated whenever it grows over 100k in size, and the  old  logs  files
       are  mailed  (uncompressed) to www@my.org after going through 5 rotations, rather than being removed. The sharedscripts means that the postrotate script will only be run once (after the
       old logs have been compressed), not once for each log which is rotated. Note that the double quotes around the first filename at the beginning of this section allows logrotate to rotate
       logs with spaces in the name. Normal shell quoting rules apply, with ', ", and \ characters supported.

       The  next  section  defines  the parameters for all of the files in /var/log/news. Each file is rotated on a monthly basis.  This is considered a single rotation directive and if errors
       occur for more than one file, the log files are not compressed.

       The last section uses tilde expansion to rotate log files in the home directory of the current user. This is only available, if your glob library supports tilde expansion. GNU glob does
       support this.

       Please  use  wildcards  with  caution.  If you specify *, logrotate will rotate all files, including previously rotated ones.  A way around this is to use the olddir directive or a more
       exact wildcard (such as *.log).

       Here is more information on the directives which may be included in a logrotate configuration file:

       compress
              Old versions of log files are compressed with gzip(1) by default. See also nocompress.

       compresscmd
              Specifies which command to use to compress log files.  The default is gzip.  See also compress.

       uncompresscmd
              Specifies which command to use to uncompress log files.  The default is gunzip.

       compressext
              Specifies which extension to use on compressed logfiles, if compression is enabled.  The default follows that of the configured compression command.

       compressoptions
              Command line options may be passed to the compression program, if one is in use.  The default, for gzip(1), is "-6" (biased towards high compression at the expense of speed).  If
              you use a different compression command, you may need to change the compressoptions to match.

       copy   Make  a copy of the log file, but don't change the original at all.  This option can be used, for instance, to make a snapshot of the current log file, or when some other utility
              needs to truncate or parse the file.  When this option is used, the create option will have no effect, as the old log file stays in place.

       copytruncate
              Truncate the original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one.  It can be used when some program  cannot  be
              told  to  close its logfile and thus might continue writing (appending) to the previous log file forever.  Note that there is a very small time slice between copying the file and
              truncating it, so some logging data might be lost.  When this option is used, the create option will have no effect, as the old log file stays in place.

       create mode owner group, create owner group
              Immediately after rotation (before the postrotate script is run) the log file is created (with the same name as the log file just rotated).  mode specifies the mode for  the  log
              file  in  octal  (the  same  as chmod(2)), owner specifies the user name who will own the log file, and group specifies the group the log file will belong to. Any of the log file
              attributes may be omitted, in which case those attributes for the new file will use the same values as the original log file for the omitted attributes. This option can  be  dis‐
              abled using the nocreate option.

       daily  Log files are rotated every day.

       dateext
              Archive  old versions of log files adding a date extension like YYYYMMDD instead of simply adding a number. The extension may be configured using the dateformat and dateyesterday
              options.

       dateformat format_string
              Specify the extension for dateext using the notation similar to strftime(3) function. Only %Y %m %d and %s specifiers are allowed.  The default value is -%Y%m%d. Note  that  also
              the  character  separating  log  name from the extension is part of the dateformat string. The system clock must be set past Sep 9th 2001 for %s to work correctly.  Note that the
              datestamps generated by this format must be lexically sortable (i.e., first the year, then the month then the day. e.g., 2001/12/01 is ok, but 01/12/2001 is not, since 01/11/2002
              would  sort  lower  while  it  is  later).  This is because when using the rotate option, logrotate sorts all rotated filenames to find out which logfiles are older and should be
              removed.

       dateyesterday
              Use yesterday's instead of today's date to create the dateext extension, so that the rotated log file has a date in its name that is the same as the timestamps within it.

       delaycompress
              Postpone compression of the previous log file to the next rotation cycle.  This only has effect when used in combination with compress.  It can be used when some  program  cannot
              be told to close its logfile and thus might continue writing to the previous log file for some time.

       extension ext
              Log  files  with ext extension can keep it after the rotation.  If compression  is  used,  the compression extension (normally .gz) appears after ext. For example you have a log‐
              file named mylog.foo and want to rotate it to mylog.1.foo.gz instead of mylog.foo.1.gz.

       hourly Log files are rotated every hour. Note that usually logrotate is configured to be run by cron daily. You have to change this configuration and run logrotate hourly to be able  to
              really rotate logs hourly.

       ifempty
              Rotate the log file even if it is empty, overriding the notifempty option (ifempty is the default).

       include file_or_directory
              Reads  the  file  given  as an argument as if it was included inline where the include directive appears. If a directory is given, most of the files in that directory are read in
              alphabetic order before processing of the including file continues. The only files which are ignored are files which are not regular files (such as directories and  named  pipes)
              and files whose names end with one of the taboo extensions, as specified by the tabooext directive.

       mail address
              When a log is rotated out-of-existence, it is mailed to address. If no mail should be generated by a particular log, the nomail directive may be used.

       mailfirst
              When using the mail command, mail the just-rotated file, instead of the about-to-expire file.

       maillast
              When using the mail command, mail the about-to-expire file, instead of the just-rotated file (this is the default).

       maxage count
              Remove rotated logs older than <count> days. The age is only checked if the logfile is to be rotated. The files are mailed to the configured address if maillast and mail are con‐
              figured.

       maxsize size
              Log files are rotated when they grow bigger than size bytes even before the additionally specified time interval (daily, weekly, monthly, or yearly).  The related size option  is
              similar  except  that  it  is mutually exclusive with the time interval options, and it causes log files to be rotated without regard for the last rotation time.  When maxsize is
              used, both the size and timestamp of a log file are considered.

       minsize  size
              Log files are rotated when they grow bigger than size bytes, but not before the additionally specified time interval (daily, weekly, monthly, or yearly).  The related size option
              is  similar except that it is mutually exclusive with the time interval options, and it causes log files to be rotated without regard for the last rotation time.  When minsize is
              used, both the size and timestamp of a log file are considered.

       missingok
              If the log file is missing, go on to the next one without issuing an error message. See also nomissingok.

       monthly
              Log files are rotated the first time logrotate is run in a month (this is normally on the first day of the month).

       nocompress
              Old versions of log files are not compressed. See also compress.

       nocopy Do not copy the original log file and leave it in place.  (this overrides the copy option).

       nocopytruncate
              Do not truncate the original log file in place after creating a copy (this overrides the copytruncate option).

       nocreate
              New log files are not created (this overrides the create option).

       nodelaycompress
              Do not postpone compression of the previous log file to the next rotation cycle (this overrides the delaycompress option).

       nodateext
              Do not archive  old versions of log files with date extension (this overrides the dateext option).

       nomail Don't mail old log files to any address.

       nomissingok
              If a log file does not exist, issue an error. This is the default.

       noolddir
              Logs are rotated in the same directory the log normally resides in (this overrides the olddir option).

       nosharedscripts
              Run prerotate and postrotate scripts for every log file which is rotated (this is the default, and overrides the sharedscripts option). The absolute  path  to  the  log  file  is
              passed as first argument to the script. If the scripts exit with error, the remaining actions will not be executed for the affected log only.

       noshred
              Do not use shred when deleting old log files. See also shred.

       notifempty
              Do not rotate the log if it is empty (this overrides the ifempty option).

       olddir directory
              Logs  are moved into directory for rotation. The directory must be on the same physical device as the log file being rotated, and is assumed to be relative to the directory hold‐
              ing the log file unless an absolute path name is specified. When this option is used all old versions of the log end up in directory.  This option may be overridden by the noold‐
              dir option.

       postrotate/endscript
              The  lines  between  postrotate  and endscript (both of which must appear on lines by themselves) are executed (using /bin/sh) after the log file is rotated. These directives may
              only appear inside a log file definition. Normally, the absolute path to the log file is passed as first argument to the script. If sharedscripts is specified, whole  pattern  is
              passed to the script.  See also prerotate. See sharedscripts and nosharedscripts for error handling.

       prerotate/endscript
              The  lines between prerotate and endscript (both of which must appear on lines by themselves) are executed (using /bin/sh) before the log file is rotated and only if the log will
              actually be rotated. These directives may only appear inside a log file definition. Normally, the absolute path to the log file is passed as first argument  to  the  script.   If
              sharedscripts is specified, whole pattern is passed to the script.  See also postrotate.  See sharedscripts and nosharedscripts for error handling.

       firstaction/endscript
              The  lines  between  firstaction and endscript (both of which must appear on lines by themselves) are executed (using /bin/sh) once before all log files that match the wildcarded
              pattern are rotated, before prerotate script is run and only if at least one log will actually be rotated.  These directives may only appear inside a log file  definition.  Whole
              pattern is passed to the script as first argument. If the script exits with error, no further processing is done. See also lastaction.

       lastaction/endscript
              The lines between lastaction and endscript (both of which must appear on lines by themselves) are executed (using /bin/sh) once after all log files that match the wildcarded pat‐
              tern are rotated, after postrotate script is run and only if at least one log is rotated. These directives may only appear inside a log file definition. Whole pattern  is  passed
              to the script as first argument. If the script exits with error, just an error message is shown (as this is the last action). See also firstaction.

       preremove/endscript
              The  lines  between preremove and endscript (both of which must appear on lines by themselves) are executed (using /bin/sh) once just before removal of a log file.  The logrotate
              will pass the name of file which is soon to be removed. See also firstaction.

       rotate count
              Log files are rotated count times before being removed or mailed to the address specified in a mail directive. If count is 0, old versions are removed rather than rotated.

       size size
              Log files are rotated only if they grow bigger then size bytes. If size is followed by k, the size is assumed to be in kilobytes.  If the M is used, the size is in megabytes, and
              if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G are all valid.

       sharedscripts
              Normally,  prerotate  and  postrotate  scripts are run for each log which is rotated and the absolute path to the log file is passed as first argument to the script. That means a
              single script may be run multiple times for log file entries which match multiple files (such as the /var/log/news/* example). If sharedscripts is specified, the scripts are only
              run  once,  no  matter how many logs match the wildcarded pattern, and whole pattern is passed to them.  However, if none of the logs in the pattern require rotating, the scripts
              will not be run at all. If the scripts exit with error, the remaining actions will not be executed for any logs. This option overrides the nosharedscripts option and implies cre‐
              ate option.

       shred  Delete log files using shred -u instead of unlink().  This should ensure that logs are not readable after their scheduled deletion; this is off by default.  See also noshred.

       shredcycles count
              Asks GNU shred(1) to overwrite log files count times before deletion.  Without this option, shred's default will be used.

       start count
              This  is  the number to use as the base for rotation. For example, if you specify 0, the logs will be created with a .0 extension as they are rotated from the original log files.
              If you specify 9, log files will be created with a .9, skipping 0-8.  Files will still be rotated the number of times specified with the rotate directive.

       su user group
              Rotate log files set under this user and group instead of using default user/group (usually root). user specifies the user name used for rotation and group  specifies  the  group
              used for rotation.

       tabooext [+] list
              The  current taboo extension list is changed (see the include directive for information on the taboo extensions). If a + precedes the list of extensions, the current taboo exten‐
              sion list is augmented, otherwise it is replaced. At startup, the taboo extension list contains .rpmsave, .rpmorig, ~,  .disabled,  .dpkg-old,  .dpkg-dist,  .dpkg-new,  .cfsaved,
              .ucf-old, .ucf-dist, .ucf-new, .rpmnew, .swp, .cfsaved, .rhn-cfg-tmp-*

       weekly Log files are rotated if the current weekday is less than the weekday of the last rotation or if more than a week has passed since the last rotation. This is normally the same as
              rotating logs on the first day of the week, but it works better if logrotate is not run every night.

       yearly Log files are rotated if the current year is not the same as the last rotation.

FILES
       /var/lib/logrotate.status  Default state file.
       /etc/logrotate.conf        Configuration options.

SEE ALSO
       gzip(1)

       <http://fedorahosted.org/logrotate/>

AUTHORS
       Erik Troan, Preston Brown, Jan Kaluza.

       <logrotate-owner@fedoraproject.org>

1.3 查看logrotate默认执行时间

logrotate切割时间默认是在每天的3:22。这个时间点可以通过crontab配置文件查看到。如下:

[root@flume-2 ~]# cat /etc/anacrontab
# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.

SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22

#period in days   delay in minutes   job-identifier   command
1       5       cron.daily              nice run-parts /etc/cron.daily
7       25      cron.weekly             nice run-parts /etc/cron.weekly
@monthly 45     cron.monthly            nice run-parts /etc/cron.monthly

如果你想在指定时间点,让logrotate切割日志的话,可以修改此配置文件的START_HOURS_RANGE参数。

注意:有人说logrotate没有按照指定的时间进行切割日志,本人没测试,一天切一次满足需求就行。

二、使用shell脚本切割

使用shell脚本切割nginx日志很简单,shell脚本内容如下:

vim /usr/local/cut_del_logs.sh

#!/bin/bash

#初始化

LOGS_PATH=/var/local/nginx

YESTERDAY=$(date -d "yesterday" +%Y%m%d)

#按天切割日志

mv ${LOGS_PATH}/access.log ${LOGS_PATH}/access_${YESTERDAY}.log

#向nginx主进程发送USR1信号,重新打开日志文件,否则会继续往mv后的文件写数据的。原因在于:linux系统中,内核是根据文件描述符来找文件的。如果不这样操作导致日志切割失败。

kill -USR1 `ps axu | grep "nginx: master process" | grep -v grep | awk '{print $2}'`

#删除7天前的日志

cd ${LOGS_PATH}

find . -mtime +7 -name "*20[1-9][3-9]*" | xargs rm -f

#或者

#find . -mtime +7 -name "access_*" | xargs rm -f

exit 0

该shell脚本有两个功能,第一个是切割nginx日志,第二个是删除7天之前的nginx日志。

在切割nginx日志的功能中,我们要注意该shell脚本命名切割的日志是以切割时,是以前一天的时间就行命名该日志文件的。

所以我们在把该shell脚本放在crontab中执行时,建议在每天的0点0分执行。如下:

vim /etc/crontab

0 0 * * * root /bin/sh /usr/local/cut_del_logs.sh

 

注意:

在没有执行 kill -USR1 `ps axu | grep "nginx: master process" | grep -v grep | awk '{print $2}'`之前,

即便已经对文件执行了mv命令而改变了文件名称,nginx还是会向新命名的文件照常写入日志数据的。

 

原因在于:linux系统中,内核是根据文件描述符来找文件的。文件描述符是linux内核为每个打开的文件命名的一个整数标识。

linux内核为每一个进程生成(或者说维护)一个”文件描述符表”,这个文件描述符表记录的是“此进程所打开的文件(进行标识)”。

在这里的环境中,nginx就是一个运行中的进程,这个进程早就打开了一个日志文件,在文件描述符表是记录了文件的。

即便日志文件的路径改变了,但是还是能够找到(根据文件描述符表可以定位)。

 ----------------------------------------------

在linux系统中,linux是通过信号与”正在运行的进程”进行通信的。linux系统中,也很多预定义好的信号,像SIGHUP。

USR1是用户自定义信号。可以理解为:进程自己定义接到这个信号该干嘛(也就是进程编写者自己确定收到这个信号干嘛还是什么都不做都行,完全交给开发人员自己决定)。

而在nginx中,它自己编写了代码处理当我接到USR1信号的时候,让nginx重新打开日志文件。具体原理如下:

1、nginx 的主进程收到USR1信号,会重新打开日志文件(以nginx配置文件中的日志名称命名,就是配置文件中access_log项所设置的值,如果文件不存在,会自动创建一个新的文件xxx.log)。

2、然后把日志文件的拥有者改为“工作进程(worker进程)”,目的是让worker进程就具备了对日志文件的读写权限(master和worker通常以不同用户运行,所以需要改变拥有者)。

3、nginx主进程会关闭重名的日志文件(也就是刚才使用mv命令重命名成xxx.log_ 20130909.log的文件),并通知工作进程使用新打开的日志文件(刚才主进程打开的文件xxx.log)。具体实现上更细化点就是,主进程把USR1信号发给worker,worker接到这个信号后,会重新打开日志文件(也就是配置文件中约定的xxx.log)

所以-USR1是Nginx的一个信号,那么还有哪些信号呢,这些信号有什么作用呢?总结如下:

主进程可以处理以下的信号:

TERM, INT 快速关闭
QUIT 从容关闭
HUP 重载配置
用新的配置开始新的工作进程
从容关闭旧的工作进程
USR1 重新打开日志文件
USR2 平滑升级可执行程序。
WINCH 从容关闭工作进程

尽管你不必自己操作工作进程,但是,它们也支持一些信号:

TERM, INT 快速关闭
QUIT 从容关闭
USR1 重新打开日志文件

 

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!