1.解决了nfs单点故障;
2.使用nfs+keepalived做了主备;
1.当nfs服务器宕机时~主备切换,依据监控keepalived主上的nfs挂了,就停掉主keepalived;依靠keepalived的nfs_check.sh监控(脚本:nfs_check.sh)
nfs_check.sh:
1.监控keepalived是否为主,为主:检查挂载是可用
2.检查此机器上的nfs是否正常,不正常,关闭keepalived
2.通过keepalived的notify_master、notify_backup、notify_fault、notify_stop的4个状态,分别对应的脚本来决定数据的同步方向
3.通过rsync+inotify实现主备间的共享目录进行同步。
VIP:192.168.109.136
master-nfs:192.168.109.137
backup-nfs:192.168.109.138
client:192.168.109.139
nfs与所有挂载机做免密安全认证,为后面判断挂载读写正常检测提供(略)
ssh-keygen
ssh-copy-id IP
一、master-nfs、backup-nfs 安装nfs服务;client安装客户端(略):参考:https://www.cnblogs.com/NGames/p/12088954.html
二、master-nfs、backup-nfs在双nfs主机上安装keepalived
yum -y install keepalived
修改配置文件:2台一样(BACKUP),主主
vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
router_id nfs
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_script chk_nfs {
script "/etc/keepalived/nfs_check.sh >> /etc/keepalived/logs/check.log"
interval 3
fall 3
rise 3
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
chk_nfs
}
virtual_ipaddress {
192.168.109.136
}
notify_master /etc/keepalived/nfs_master.sh
notify_backup /etc/keepalived/nfs_backup.sh
notify_fault /etc/keepalived/nfs_fault.sh
notify_stop /etc/keepalived/nfs_stop.sh
}
mkdir -pv /etc/keepalived/logs
---------------------------------------------------------------
vim nfs_backup.sh
vip=192.168.109.136
NUM=`ip a | grep $vip | wc -l`
LOGFILE="/etc/keepalived/logs/keepalived-nfs-state.log"
echo "[slave]" >> $LOGFILE
date >> $LOGFILE
echo "Being slave...." >> $LOGFILE 2>&1
if [ $NUM -eq 1 ]
then
echo "$vip from nfs_backup.sh 开始启用自动同步脚本" >> $LOGFILE 2>&1
nohup bash /scripts/inotifi.sh &
else
echo "无VIP from nfs_backup.sh 停止自动同步脚本" >> $LOGFILE 2>&1
kill -9 `ps aux | grep inotifi.sh | grep -v grep | awk '{print $2}'`
kill -9 `ps aux | grep inotifywait | grep -v grep | awk '{print $2}'`
fi
vim nfs_check.sh
VIP=192.168.109.136
ip=`/sbin/ip a|grep "ens33"|grep "$VIP"`
share_dir=/apprun/nfsdata
nfs_ceshi_file=/apprun/nfsdata/a.txt
if [ "$ip" != "" ];then
for i in `cat /etc/keepalived/web_IP`;
do echo ${i};
TXT=`/usr/bin/ssh ${i} "cat $nfs_ceshi_file"`
if [ "$TXT" != "NFS server OK..." ];then
/usr/bin/ssh ${i} "umount -l $share_dir && mount -a"
fi
done
fi
port=`netstat -ntlp | grep 2049`
if [ "$port" == "" ];then
systemctl stop keepalived
fi
echo "NFS server OK..." > /apprun/nfsdata/a.txt
vim nfs_fault.sh
vip=192.168.109.136
NUM=`ip a | grep $vip | wc -l`
LOGFILE="/etc/keepalived/logs/keepalived-nfs-state.log"
echo "[slave]" >> $LOGFILE
date >> $LOGFILE
echo "Being slave...." >> $LOGFILE 2>&1
if [ $NUM -eq 1 ]
then
echo "$vip from nfs_fault.sh 开始启用自动同步脚本" >> $LOGFILE 2>&1
nohup bash /scripts/inotifi.sh &
else
echo "无VIP from nfs_fault.sh 停止自动同步脚本" >> $LOGFILE 2>&1
kill -9 `ps aux | grep inotifi.sh | grep -v grep | awk '{print $2}'`
kill -9 `ps aux | grep inotifywait | grep -v grep | awk '{print $2}'`
fi
vim nfs_master.sh
vip=192.168.109.136
NUM=`ip a | grep $vip | wc -l`
LOGFILE="/etc/keepalived/logs/keepalived-nfs-state.log"
echo "[master]" >> $LOGFILE
date >> $LOGFILE
echo "Being master...." >> $LOGFILE 2>&1
sleep 1
if [ $NUM -eq 1 ]
then
echo "$vip from nfs_master.sh 开始启用自动同步脚本" >> $LOGFILE 2>&1
nohup bash /scripts/inotifi.sh &
else
echo "无VIP from nfs_master.sh 停止自动同步脚本" >> $LOGFILE 2>&1
kill -9 `ps aux | grep inotifi.sh | grep -v grep | awk '{print $2}'`
kill -9 `ps aux | grep inotifywait | grep -v grep | awk '{print $2}'`
fi
vim nfs_stop.sh
vip=192.168.109.136
NUM=`ip a | grep $vip | wc -l`
LOGFILE="/etc/keepalived/logs/keepalived-nfs-state.log"
echo "[slave]" >> $LOGFILE
date >> $LOGFILE
echo "Being slave...." >> $LOGFILE 2>&1
if [ $NUM -eq 1 ]
then
echo "$vip from nfs_stop.sh 开始启用自动同步脚本" >> $LOGFILE 2>&1
nohup bash /scripts/inotifi.sh &
else
echo "无VIP from nfs_stop.sh 停止自动同步脚本" >> $LOGFILE 2>&1
kill -9 `ps aux | grep inotifi.sh | grep -v grep | awk '{print $2}'`
kill -9 `ps aux | grep inotifywait | grep -v grep | awk '{print $2}'`
fi
web_IP 记录需要挂载的机器ip
-------------------------------------------------------------------------------------------
配置文件同步:
rsync -avz --progress -e ssh /etc/keepalived/* 192.168.109.138:/etc/keepalived/
1.此时启动keepalived会因没有同步数据脚本/scripts/inotifi.sh ,可以先配置下面的数据同步
2.要先nfs,不然检测会关闭keepalived
systemctl restart keepalived.service
systemctl enable keepalived.service
=====================================================================================================
三、创建共享目录
mkdir -pv /apprun/nfsdata
四、共享目录资源同步
1.情况一:master_nfs为(192.168.109.137);backup_nfs为(192.168.109.138)
安装:backup_nfs为(192.168.109.138)
\\安装rsync
yum -y install rsync
vim /etc/rsyncd.conf
pidfile = /var/run/rsyncd.pid
lock file = /var/run/rsyncd.lock
log file = /var/log/rsyncd.log
[backup]
path = /apprun/nfsdata
comment = client(192.168.109.138)
uid = root
gid = root
port = 873
ignore errors
use chroot = no
read only = no
list = no
max connections = 200
timeout = 600
auth users = wangbocheng
secrets file = /etc/rsync.pass
\\创建用户认证文件
echo "wangbocheng:wang123" > /etc/rsync.pass
chmod 600 /etc/rsync.pass
\\启动
systemctl restart rsyncd
systemctl enable rsyncd
---------------------------------------------------
master_nfs为(192.168.109.137)
yum -y install rsync inotify-tools
依据系统版本选择源:
wget http://mirrors.aliyun.com/repo/epel-7.repo
yum -y install inotify-tools
echo "wang123" > /etc/rsync.pass
chmod 600 /etc/rsync.pass
手动同步命令:(手动数据同步)
rsync -avzp --port=873 --progress --delete /apprun/nfsdata/ wangbocheng@192.168.109.138::backup --password-file=/etc/rsync.pass
自动同步
\\编写脚本:vim /scripts/inotifi.sh
host=192.168.109.138
src=/apprun/nfsdata/
des=backup
password=/etc/rsync.pass
user=wangbocheng
inotifywait=/usr/bin/inotifywait
$inotifywait -mrq --timefmt '%Y%m%d %H:%M' --format '%T %w%f%e' -e modify,delete,create,attrib $src | while read files ;do
rsync -avzP --delete --timeout=100 --password-file=${password} $src $user@$host::$des
echo "${files} was rsynced" >>/tmp/rsync.log 2>&1
done
//启动脚本
nohup bash /scripts/inotifi.sh &
\\查看是否同步
在master-nfs共享目录下写入文件,查看日志,查看 backup-nfs目录同步情况
######设置脚本的开机自启动 (依各人环境需求是否配置,此处无需)
######touch /etc/rc.d/rc.local
######chmod +x /etc/rc.d/rc.local
######echo "nohup /bin/bash /scripts/inotify.sh" >> /etc/rc.d/rc.local
######重启后查看
2.情况二:master_nfs为(192.168.109.138);backup_nfs为(192.168.109.137)
安装:backup_nfs为(192.168.109.137)
\\安装rsync
yum -y install rsync
vim /etc/rsyncd.conf
pidfile = /var/run/rsyncd.pid
lock file = /var/run/rsyncd.lock
log file = /var/log/rsyncd.log
[backup]
path = /apprun/nfsdata
comment = client(M138)
uid = root
gid = root
port = 873
ignore errors
use chroot = no
read only = no
list = no
max connections = 200
timeout = 600
auth users = wangbocheng2
secrets file = /etc/rsync2.pass
\\创建用户认证文件
echo "wangbocheng2:wang123" > /etc/rsync2.pass
chmod 600 /etc/rsync2.pass
\\启动
systemctl restart rsyncd
systemctl enable rsyncd
---------------------------------------------------
master_nfs为(192.168.109.138)
yum -y install rsync inotify-tools
依据系统版本选择源:
wget http://mirrors.aliyun.com/repo/epel-7.repo
yum -y install inotify-tools
echo "wang123" > /etc/rsync2.pass
chmod 600 /etc/rsync2.pass
手动同步命令:(手动数据同步)
rsync -avzp --port=873 --progress --delete /apprun/nfsdata/ wangbocheng2@192.168.109.137::backup --password-file=/etc/rsync2.pass
自动同步
\\编写脚本:vim /scripts/inotifi.sh
host=192.168.109.137
src=/apprun/nfsdata/
des=backup
password=/etc/rsync2.pass
user=wangbocheng2
inotifywait=/usr/bin/inotifywait
$inotifywait -mrq --timefmt '%Y%m%d %H:%M' --format '%T %w%f%e' -e modify,delete,create,attrib $src | while read files ;do
rsync -avzP --delete --timeout=100 --password-file=${password} $src $user@$host::$des
echo "${files} was rsynced" >>/tmp/rsync.log 2>&1
done
//启动脚本(1.2情况只能是一边启动同步脚本,不然数据混乱)
nohup bash /scripts/inotifi.sh &
\\查看是否同步
在master-nfs共享目录下写入文件,查看日志,查看 backup-nfs目录同步情况
===============================================================================
nfs服务安装(略)参考:https://www.cnblogs.com/NGames/p/12088954.html
日志:
keepalived 状态切换日志相关记录:tail -f /etc/keepalived/logs/keepalived-nfs-state.log
健康检查相关日志记录:tail -f /etc/keepalived/logs/check.log
数据源同步相关日志记录:tail -f /tmp/rsync.log
==============================================
已验证,
情况1:nfs意外关闭;状态切换正常,数据同步正常
情况2:keepalived意外关闭;状态切换正常,数据同步正常
来源:https://www.cnblogs.com/NGames/p/12181056.html