【K8s+Docker技术全解】08.Master主控节点服务-keepalived配置vip

## 99.101/102 keepalived配置vip 配置keepalived实现nginx高可用，不能因为一个nginx宕机造成服务不可用。 ### 99.101配置keepalived主 #### 安装keepalived ```bash # 192.168.99.101 [root@k8s99-101 ~]# yum install keepalived -y ``` #### 监听端口脚本check_port.sh ```bash # 192.168.99.101 [root@k8s99-101 ~]# vim /etc/keepalived/check_port.sh # 写入以下内容 #!/bin/bash # keepalived监听nginx端口脚本 # 使用方法，在keepalived配置文件中 # vrrp_script chk_nginx { # script "/etc/keepalived/check_port.sh 7443" # .... CHK_PORT=$1 if [ -n "$CHK_PORT" ];then PORT_PROCESS=`ss -lnt | grep $CHK_PORT | wc -l` if [ $PORT_PROCESS -eq 0 ];then echo "Port $CHK_PORT is not used, please check." exit 1 fi else echo "Port $CHK_PORT is used, service is normal." fi ``` 这个脚本表示，如果nginx的7443端口没被监听了，表示nginx服务挂了，退出代码为1。 添加执行权限 ```bash # 192.168.99.101 [root@k8s99-101 ~]# chmod +x /etc/keepalived/check_port.sh # 测试这个脚本能否正常使用 [root@k8s99-101 ~]# /etc/keepalived/check_port.sh Port is used, service is normal. ``` #### 主keepalived配置 主和备的配置文件还是有所区别的 ```bash # 192.168.99.101 [root@k8s99-101 ~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak [root@k8s99-101 ~]# vim /etc/keepalived/keepalived.conf # 添加下面的内容到conf文件中 global_defs { # (VI_1) the virtual router id must be set，运行keepalived的机器的一个标识，通常可设为hostname。故障发生时，发邮件时显示在邮件主题中的信息 router_id chk_port_in_101 } vrrp_script chk_nginx { # 指定监控脚本，检测nginx服务是否正常运行 script "/etc/keepalived/check_port.sh 7443" # 指定监控时间，每2s执行一次 interval 2 # 脚本结果导致的优先级变更，检测失败（脚本返回非0）则优先级 -20 weight -20 } # keepalived在同一virtual_router_id中priority（0-255）最大的会成为master，也就是接管VIP，当priority最大的主机发生故障后，priority较小将会接管。 vrrp_instance VI_1 { # 指定keepalived的角色，主机设置为MASTER，备用机设置为BACKUP，注意这里的state指定instance(Initial)的初始状态， # 就是说在配置好后，这台服务器的初始状态就是这里指定的，但这里指定的不算，还是得要通过竞选通过优先级来确定。 # 如果这里设置为MASTER，但如若他的优先级不及另外一台，那么这台在发送通告时，会发送自己的优先级，另外一台发现优先级不如自己的高，那么他会就回抢占为MASTER。 state MASTER # 指定HA监测网络的接口。centos7使用 ip addr 获取 interface ens33 # 发送多播数据包时的源IP地址，这里实际上就是在哪个地址上发送VRRP通告，这个非常重要，一定要选择稳定的网卡端口来发送，这里相当于heartbeat的心跳端口， # 如果没有设置那么就用默认的绑定的网卡的IP，也就是interface指定的IP地址 mcast_src_ip 192.168.99.101 # 主备的virtual_router_id必须一样，可以设置为IP后一组：must be between 1 & 255 virtual_router_id 101 # 优先级值，在同一个vrrp_instance下， MASTRE 一定要高于 BAUCKUP，MASTER恢复后，BACKUP自动交接 priority 100 # VRRP 广播周期秒数，如果没检测到该广播，就被认为服务挂了，将切换主备 advert_int 1 # 设置为不抢占。默认是抢占的，当高优先级的机器恢复后，会抢占低优先级的机器成为MASTER，而不抢占，则允许低优先级的机器继续成为MASTER， # 即使高优先级的机器已经上线。如果要使用这个功能，则初始化状态必须为BACKUP。 nopreempt # 设置验证类型和密码。主从必须一样 authentication { # 设置vrrp验证类型，主要有PASS和AH两种 auth_type PASS # 加密的密码，两台服务器一定要一样，才能正常通信 auth_pass passwd } track_script { # 执行监控的服务，引用VRRP脚本，即在 vrrp_script 部分指定的名字。定期运行它们来改变优先级 chk_nginx } virtual_ipaddress { # VRRP HA 虚拟地址 如果有多个VIP，继续换行填写 192.168.99.100 } } ``` 注意： `state`主设置为`MASTER`，备设置为`BACKUP` `priority`主的只一定要大于备 `nopreempt`设置为不抢占模式 #### 防火墙允许VRRP路由冗余协议 否则主机之前keepalived无法正常通信 ```baswh [root@k8s99-101 ~]# firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT success [root@k8s99-101 ~]# firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT success [root@k8s99-101 ~]# firewall-cmd --reload success ``` 允许防火墙之后马上就在messages日志中看到keepalived的通信日志了，否则他们之前无法正常通信，vip在各自的主机上都会存在。 #### 启动keepalived ```bash [root@k8s99-101 ~]# systemctl start keepalived [root@k8s99-101 ~]# systemctl status keepalived [root@k8s99-101 ~]# systemctl enable keepalived ``` #### 检查vip所在的网卡 ```bash [root@k8s99-101 ~]# ip a show ens33 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:18:a0:76 brd ff:ff:ff:ff:ff:ff inet 192.168.99.101/24 brd 192.168.99.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.99.100/32 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::b3ad:c1c9:36cb:8198/64 scope link noprefixroute valid_lft forever preferred_lft forever ``` 可以通过`tail -f /var/log/messages`查到keepalived的日志。 ### 99.102配置keepalived备 ```bash # 192.168.99.102 # 安装keepalived yum install keepalived -y # 编写检测脚本 vim /etc/keepalived/check_port.sh #!/bin/bash # keepalived监听nginx端口脚本 # 使用方法，在keepalived配置文件中 # vrrp_script chk_nginx { # script "/etc/keepalived/check_port.sh 7443" # .... CHK_PORT=$1 if [ -n "$CHK_PORT" ];then PORT_PROCESS=`ss -lnt | grep $CHK_PORT | wc -l` if [ $PORT_PROCESS -eq 0 ];then echo "Port $CHK_PORT is not used, please check." exit 1 fi else echo "Port $CHK_PORT is used, service is normal." fi # 添加执行权限并尝试执行 chmod +x /etc/keepalived/check_port.sh /etc/keepalived/check_port.sh # 备份原来的keepalived配置 mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak # 创建新的keepalived配置 cat > /etc/keepalived/keepalived.conf << EOF global_defs { router_id chk_port_in_101 } vrrp_script chk_nginx { script "/etc/keepalived/check_port.sh 7443" interval 2 weight -20 } vrrp_instance VI_1 { state BACKUP interface ens33 mcast_src_ip 192.168.99.102 virtual_router_id 101 # 权重值 MASTRE 一定要高于 BAUCKUP priority 90 advert_int 1 authentication { auth_type PASS auth_pass passwd } track_script { chk_nginx } virtual_ipaddress { # VIP 虚拟IP地址 192.168.99.100 } } EOF # CentOS防火墙允许keepalived通信 firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT firewall-cmd --reload # 启动keepalived systemctl start keepalived systemctl status keepalived systemctl enable keepalived # 查看网卡信息，这是BACKUP，只要MASTER正常，vip不会漂移过来（前提是主备的keepalived通信是正常的） ip a show ens33 | grep 99.100 # 查看keepalived日志 tail -f /var/log/messages ``` ### 检查测试keepalived功能 #### 正常运行，查看主的状态 ```bash # 192.168.99.101 # 查看nginx运行状态 [root@k8s99-101 ~]# netstat -luntp | grep 7443 # 查看vip状态 [root@k8s99-101 ~]# ip a | grep 99.100 inet 192.168.99.100/32 scope global ens33 ``` 两者都正常运行时，vip在MASTER上 #### 正常运行，查看备的状态 ```bash # 192.168.99.102 [root@k8s99-102 ~]# netstat -luntp | grep 7443 tcp 0 0 0.0.0.0:7443 0.0.0.0:* LISTEN 1099/nginx: master [root@k8s99-102 ~]# ip a | grep 99.115 ``` #### 模拟nginx宕机，查看主的状态 ```bash # 192.168.99.101 [root@k8s99-101 ~]# nginx -s stop # vip已经漂移走了 [root@k8s99-101 ~]# ip a | grep 99.100 [root@k8s99-101 ~]# netstat -luntp | grep 7443 # 查看主的日志 [root@k8s99-101 ~]# tail -f /var/log/messages Jun 1 23:36:46 k8s99-101 Keepalived_vrrp[3572]: /etc/keepalived/check_port.sh 7443 exited with status 1 Jun 1 23:36:46 k8s99-101 Keepalived_vrrp[3572]: VRRP_Script(chk_nginx) failed Jun 1 23:36:46 k8s99-101 Keepalived_vrrp[3572]: VRRP_Instance(VI_1) Changing effective priority from 100 to 80 Jun 1 23:36:47 k8s99-101 Keepalived_vrrp[3572]: VRRP_Instance(VI_1) Received advert with higher priority 90, ours 80 Jun 1 23:36:47 k8s99-101 Keepalived_vrrp[3572]: VRRP_Instance(VI_1) Entering BACKUP STATE Jun 1 23:36:47 k8s99-101 Keepalived_vrrp[3572]: VRRP_Instance(VI_1) removing protocol VIPs. Jun 1 23:36:48 k8s99-101 Keepalived_vrrp[3572]: /etc/keepalived/check_port.sh 7443 exited with status 1 Jun 1 23:36:50 k8s99-101 Keepalived_vrrp[3572]: /etc/keepalived/check_port.sh 7443 exited with status 1 ``` #### 模拟nginx宕机，查看备的状态 ```bash # 192.168.99.102 # vip漂移到备上 [root@k8s99-102 ~]# ip a | grep 99.100 inet 192.168.99.100/32 scope global ens33 [root@k8s99-102 ~]# tail -f /var/log/messages Jun 1 23:35:12 k8s99-102 systemd: Reloading. Jun 1 23:36:47 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) forcing a new MASTER election Jun 1 23:36:48 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) Transition to MASTER STATE Jun 1 23:36:49 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) Entering MASTER STATE Jun 1 23:36:49 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) setting protocol VIPs. Jun 1 23:36:49 k8s99-102 Keepalived_vrrp[2038]: Sending gratuitous ARP on ens33 for 192.168.99.100 Jun 1 23:36:49 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.99.100 ``` 可以看到主k8s99-101上的vip就没有了，漂移到k8s99-102上面 #### 模拟主恢复，查看主的状态 当再次启动k8s99-101的nginx ```bash # 192.168.99.101 [root@k8s99-101 ~]# nginx [root@k8s99-101 ~]# netstat -luntp | grep 7443 tcp 0 0 0.0.0.0:7443 0.0.0.0:* LISTEN 16121/nginx: master # vip仍然没有漂移回来，因为配置了 [root@k8s99-101 ~]# ip a show ens33 | grep 99.100 # 查看日志 [root@k8s99-102 ~]# tail -f /var/log/messages Apr 25 22:56:14 k8s99-101 Keepalived_vrrp[14189]: /etc/keepalived/check_port.sh 7443 exited with status 1 Apr 25 22:56:16 k8s99-101 Keepalived_vrrp[14189]: VRRP_Script(chk_nginx) succeeded Apr 25 22:56:16 k8s99-101 Keepalived_vrrp[14189]: VRRP_Instance(VI_1) Changing effective priority from 80 to 100 ``` 但vip仍然没有回来，因为在主keepalived配置`vrrp_instance VI_1`中有一个`nopreempt`配置，即为非抢占式，可能会因为一些网络波动造成keepalived判断错误，vip漂移了。在**生产环境中vip不能随意动**的，就算主服务恢复正常后，vip也不能随意漂移回来。如果要让vip重新回来，需要**人工确认**好本机的服务完全正常才行。备上vip以及日志均无变化。 #### 主人工确认恢复，重启主的keepalived 在k8s99-101上进行人工确认 ```bash # 192.168.99.101 [root@k8s99-101 ~]# netstat -luntp | grep 7443 tcp 0 0 0.0.0.0:7443 0.0.0.0:* LISTEN 16121/nginx: master # 确实起来了，然后再重启keepalived [root@k8s99-101 ~]# systemctl restart keepalived # 查看日志 [root@k8s99-101 ~]# tail -f /var/log/messages Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: (VI_1): Warning - nopreempt will not work with initial state MASTER Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: SECURITY VIOLATION - scripts are being executed but script_security not enabled. Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: VRRP_Instance(VI_1) removing protocol VIPs. Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: Using LinkWatch kernel netlink reflector... Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)] Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: VRRP_Script(chk_nginx) succeeded Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: VRRP_Instance(VI_1) Transition to MASTER STATE Jun 1 23:42:33 k8s99-101 Keepalived_vrrp[5661]: VRRP_Instance(VI_1) Entering MASTER STATE Jun 1 23:42:33 k8s99-101 Keepalived_vrrp[5661]: VRRP_Instance(VI_1) setting protocol VIPs. Jun 1 23:42:33 k8s99-101 Keepalived_vrrp[5661]: Sending gratuitous ARP on ens33 for 192.168.99.100 Jun 1 23:42:33 k8s99-101 Keepalived_vrrp[5661]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.99.100 # vip就回来了 [root@k8s99-101 ~]# ip a show ens33 | grep 99.100 inet 192.168.99.100/32 scope global ens33 ``` #### 查看备状态 再查看备的状态 ```bash # 192.168.99.102 [root@k8s99-102 ~]# tail -f /var/log/messages Jun 1 23:42:32 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) Received advert with higher priority 100, ours 90 Jun 1 23:42:32 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) Entering BACKUP STATE Jun 1 23:42:32 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) removing protocol VIPs. [root@k8s99-102 ~]# ip a | grep 99.115 # vip漂移到主上了 ```

明人不说暗话 领红包就是支持！！！ 来都来了，再到页面底部看下呗~ ↓ ↓ ↓

本文地址： http://blog.starmeow.cn/detail/63923f49e5241343aa7acb6a06a751e7/

很赞哦！ (2)