实验背景
A文件服务器的/opt/dataA 和 B文件服务器的/opt/dataB目录实时同步
A正常时，A为主，B为备，B去实时同步A
A挂了时，B为主，待A起来后，A为备，去实时同步B，并且能补上挂掉期间B增加的数据，反之亦然
keepalived + rsync +inotify 实现真正的高效数据实时同步
https://fandenggui.com/post/keepalived-rsync-inotify.html
真正的inotify+rsync实时同步 彻底告别同步慢
http://www.ttlsa.com/web/let-infotify-rsync-fast
一、实验环境
操作系统：CentOS7.2
serverA： 192.168.1.102
serverB： 192.168.1.103
vip： 192.168.1.120
二、软件安装
在 serverA和serverB上
# yum -y install epel-release
# yum -y install keepalived inotity-tools rsync httpd
# systemctl enable httpd keepalived
rsync-inotify.service 会由后面的Keepalived程序根据主备角色变化情况，自动的运行和关闭
设置selinux为宽松模式
# setenforce 0
# sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config
关闭防火墙
# systemctl stop firewalld
# systemctl disable firewalld
主机互信
我们同步采用rsync的ssh模式，需要让serverA和 serverB 相互互信
三、主机配置
serverA
# vim /opt/data_realtime_rsync.sh
#################################################
#!/bin/bash
# File name:data_realtime_sync.sh
# File path: /opt/data_realtime_sync.sh
src_dir="/opt/dataA"
dest_dir="/opt/dataB"
src_ip="192.168.1.102"
dest_ip="192.168.1.103"
. /etc/init.d/functions
cd ${src_dir}
inotifywait -mrq -e modify,attrib,close_write,move,create,delete --format '%e %w%f' ./ |
while read file;do
INO_EVENT=$(echo $file | awk '{print $1}')
INO_FILE=$(echo $file | awk '{print $2}')
if [[ $INO_EVENT =~ 'CREATE' ]] || [[ $INO_EVENT =~ 'MODIFY' ]] || [[ $INO_EVENT =~ 'CLOSE_WRITE' ]] || [[ $INO_EVENT =~ 'MOVED_TO' ]];then
for ip in ${dest_ip};do
rsync -avzcR -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
done
fi
if [[ $INO_EVENT =~ 'ATTRIB' ]];then
if [ ! -d "$INO_FILE" ];then
for ip in ${dest_ip};do
rsync -avzcR -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
done
fi
fi
if [[ $INO_EVENT =~ 'DELETE' ]] || [[ $INO_EVENT =~ 'MOVED_FROM' ]];then
for ip in ${dest_ip};do
rsync -avzcR --delete -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
done
fi
done
##############################################
# vim /etc/keepalived/keepalived.conf
###########################################
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script check_httpd {
script "/etc/keepalived/check_httpd.sh"
interval 5
}
vrrp_instance VI_1 {
state BACKUP
interface eno16777736
virtual_router_id 100
priority 100
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
check_httpd
}
virtual_ipaddress {
192.168.1.120
}
notify_master "/etc/keepalived/rsync-inotify.sh start"
notify_backup "/etc/keepalived/rsync-inotify.sh stop"
notify_stop "/etc/keepalived/rsync-inotify.sh send"
}
##########################################
# vim /etc/keepalived/check_httpd.sh
##############################
#!/bin/sh
#
httpd_status1=$(ps -C httpd --no-heading|wc -l)
if [ "${httpd_status1}" = "0" ]; then
systemctl start httpd.service
sleep 5
httpd_status2=$(ps -C httpd --no-heading|wc -l)
if [ "${httpd_status2}" = "0" ]; then
systemctl stop keepalived.service
fi
fi
#############################
# vim /etc/keepalived/rsync-inotify.sh
########################
#!/bin/bash
case "$1" in
start )
systemctl start rsync-inotity.service
;;
stop )
systemctl stop rsync-inotity.service
;;
restart )
systemctl restart rsync-inotity.service
;;
send )
echo "Keepalived stoppd in 192.168.1.102" | mail -s "Alarm Mail" example@qq.com
;;
* )
echo "Usage:$0 start|stop|restart|send"
;;
esac
########################
# vim /etc/systemd/system/rsync-inotity.service
################################
[Unit]
Description=File instant rsync script
[Service]
ExecStart=/usr/bin/nohup /opt/data_realtime_rsync.sh &
[Install]
WantedBy=multi-user.target
################################
# vim /etc/full-rsync.conf
#################################
src_dir="/opt/dataA"
dest_dir="/opt/dataB"
src_ip="192.168.1.102"
dest_ip="192.168.1.103"
##################################
# vim /etc/systemd/system/full-rsync.service
####################################
[Unit]
Description=Full rsync script
After=network.target sshd.service
Wants=sshd.service
[Service]
EnvironmentFile=/etc/full-rsync.conf
ExecStart=/usr/bin/rsync -avz --delete -e ssh root@${dest_ip}:${dest_dir}/ ${src_dir}
Type=oneshot
[Install]
WantedBy=multi-user.target
#####################################
serverB
# vim /opt/data_realtime_rsync.sh
#############################
#!/bin/bash
# File name:data_realtime_sync.sh
# File path: /opt/data_realtime_sync.sh
src_dir="/opt/dataB"
dest_dir="/opt/dataA"
src_ip="192.168.1.103"
dest_ip="192.168.1.102"
. /etc/init.d/functions
cd ${src_dir}
inotifywait -mrq -e modify,attrib,close_write,move,create,delete --format '%e %w%f' ./ |
while read file;do
INO_EVENT=$(echo $file | awk '{print $1}')
INO_FILE=$(echo $file | awk '{print $2}')
if [[ $INO_EVENT =~ 'CREATE' ]] || [[ $INO_EVENT =~ 'MODIFY' ]] || [[ $INO_EVENT =~ 'CLOSE_WRITE' ]] || [[ $INO_EVENT =~ 'MOVED_TO' ]];then
for ip in ${dest_ip};do
rsync -avzcR -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
done
fi
if [[ $INO_EVENT =~ 'ATTRIB' ]];then
if [ ! -d "$INO_FILE" ];then
for ip in ${dest_ip};do
rsync -avzcR -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
done
fi
fi
if [[ $INO_EVENT =~ 'DELETE' ]] || [[ $INO_EVENT =~ 'MOVED_FROM' ]];then
for ip in ${dest_ip};do
rsync -avzcR --delete -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
done
fi
done
###################################
# vim /etc/keepalived/keepalived.conf
###########################################
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script check_httpd {
script "/etc/keepalived/check_httpd.sh"
interval 5
}
vrrp_instance VI_1 {
state BACKUP
interface eno16777736
virtual_router_id 90
priority 100
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
check_httpd
}
virtual_ipaddress {
192.168.1.120
}
notify_master "/etc/keepalived/rsync-inotify.sh start"
notify_backup "/etc/keepalived/rsync-inotify.sh stop"
notify_stop "/etc/keepalived/rsync-inotify.sh send"
}
##########################################
# vim /etc/keepalived/check_httpd.sh
##############################
#!/bin/sh
#
httpd_status1=$(ps -C httpd --no-heading|wc -l)
if [ "${httpd_status1}" = "0" ]; then
systemctl start httpd.service
sleep 5
httpd_status2=$(ps -C httpd --no-heading|wc -l)
if [ "${httpd_status2}" = "0" ]; then
systemctl stop keepalived.service
fi
fi
#############################
# vim /etc/keepalived/rsync-inotify.sh
########################
#!/bin/bash
case "$1" in
start )
systemctl start rsync-inotity.service
;;
stop )
systemctl stop rsync-inotity.service
;;
restart )
systemctl restart rsync-inotity.service
;;
send )
echo "Keepalived stoppd in 192.168.1.102" | mail -s "Alarm Mail" example@qq.com
;;
* )
echo "Usage:$0 start|stop|restart|send"
;;
esac
########################
# vim /etc/systemd/system/rsync-inotity.service
################################
[Unit]
Description=File instant rsync script
[Service]
ExecStart=/usr/bin/nohup /opt/data_realtime_rsync.sh &
[Install]
WantedBy=multi-user.target
################################
# vim /etc/full-rsync.conf
#################################
src_dir="/opt/dataB"
dest_dir="/opt/dataA"
src_ip="192.168.1.103"
dest_ip="192.168.1.102"
##################################
# vim /etc/systemd/system/full-rsync.service
####################################
[Unit]
Description=Full rsync script
After=network.target sshd.service
Wants=sshd.service
[Service]
EnvironmentFile=/etc/full-rsync.conf
ExecStart=/usr/bin/rsync -avz --delete -e ssh root@${dest_ip}:${dest_dir}/ ${src_dir}
Type=oneshot
[Install]
WantedBy=multi-user.target
#####################################
四、测试
# systemctl status httpd
# systemctl status keepalived
# systemctl status rsync-inotity
五、全量同步
手动方式
A 全量同步B
# vim full_rsync_from_B_to_A.sh
######################################################
#!/bin/bash
src_dir="/opt/dataA"
dest_dir="/opt/dataB"
src_ip="192.168.1.102"
dest_ip="192.168.1.103"
rsync -avz --delete -e ssh root@${dest_ip}:${dest_dir}/ ${src_dir}/
#######################################################
B 全量同步A
# vim full_rsync_from_A_to_B.sh
#######################################################
#!/bin/bash
src_dir="/opt/dataB"
dest_dir="/opt/dataA"
src_ip="192.168.1.103"
dest_ip="192.168.1.102"
rsync -avz --delete -e ssh root@${dest_ip}:${dest_dir}/ ${src_dir}/
#######################################################
系统服务
# systemctl daemon-reload
# systemctl start full-rsync
# systemctl status full-rsync
通过/etc/rc.d/rc.local 开机执行
# vim /etc/rc.d/rc.local
##################
# Full rsync
src_dir="/xxx"
dest_dir="/xxx"
src_ip="xx.xx.xx.xx"
dest_ip="xx.xx.xx.xx"
rsync -avz --delete -e ssh root@${dest_ip}:${dest_dir}/ ${src_dir}
##################
# chmod +x /etc/rc.d/rc.local
六、优化inotity
如果不进行优化，遇到瓶颈会有如下错误提示：
Failed to watch /your/path; upper limit on inotify watches reached!
Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify/max_user_watches'.
主要优化 max_user_watches 及 max_queued_events
具体大小可以等于当前数据盘的 inode最大值
这里以1T数据盘为例：
# echo "67108864" > /proc/sys/fs/inotify/max_user_watches
# echo "67108864" > /proc/sys/fs/inotify/max_queued_events
将下面参数写入 /etc/sysctl.conf，永久生效
# vim
/etc/sysctl.conf
fs.inotify.max_user_watches = 67108864
fs.inotify.max_queued_events = 67108864
七、参考
keepalived + rsync +inotify 实现真正的高效数据实时同步
https://fandenggui.com/post/keepalived-rsync-inotify.html
真正的inotify+rsync实时同步 彻底告别同步慢
http://www.ttlsa.com/web/let-infotify-rsync-fast
【Keepalived 】keepalived check and notify scripts
大家知道keepalived会有四种状态的变化，每种状态变化时，都可以调用一个脚本
当进入Master状态时会呼叫notify_master
当进入Backup状态时会呼叫notify_backup
当发现异常情况时进入Fault状态呼叫notify_fault
当Keepalived程序终止时则呼叫notify_stop
进入Master和Backup这两种状态很容易理解了，就是分别变为主和从
进入Fault，简单的说就是keepalived发现自己有问题了，不能能再去参与Master竞选了，你得修好他才行。
进入这种状态一般是keepalived自身出问题了，或者keepalived检测的网卡出问题不通了，再或者就是我们自己写的检测业务的脚本返回错误
进入Stop 这个就容易理解了，执行service keepalived stop 或者 systemctl stop keepalived 就会进入这个状态。
设置selinux为宽松模式
# setenforce 0
# sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config
CentOS防火墙默认是不允许keepalived使用 vrrp的组播。
如果不开启组播ip,keepalived双机不能实现热备的效果,只能实现负载的效果,即虚拟ip不能实现漂移 。
Check that the multicast IP and protocol for VRRP are allowed in the firewall on both servers.
For firewalld:
添加规则
# firewall-cmd --direct --permanent--add-rule ipv4 filterINPUT 0 --in-interface eth0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
# firewall-cmd --direct --permanent--add-ruleipv4 filterOUTPUT 0 --out-interface eth0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
# firewall-cmd --reload
删除规则
# firewall-cmd --direct --permanent--remove-ruleipv4 filter INPUT 0 --in-interface eth0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
# firewall-cmd --direct --permanent--remove-rule ipv4 filter OUTPUT 0 --out-interface eth0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
# firewall-cmd --reload
For iptables:
添加规则
# iptables -A INPUT -p vrrp -j ACCEPT
# iptables -A OUTPUT -p vrrp -j ACCEPT
# service iptables save
删除规则
# iptables -D INPUT -p vrrp -j ACCEPT
# iptables -D OUTPUT -p vrrp -j ACCEPT
# service iptables save
keepalived基本应用解析
http://blog.51cto.com/pangge/1301878
keepalived官方文档
http://www.keepalived.org/doc/introduction.html
使用keepalived实现redis的故障切换
http://peiqiang.net/2014/11/21/keepalived-and-redis.html
Kamailio High Availability Done Right with Keepalived
http://blog.unicsolution.com/2015/01/kamailio-high-availability-with.html
Keepalived Check and Notify Scripts
