您现在的位置是: 网站首页 >Docker >Docker&Kubernetes技术全解 Docker

【K8s+Docker技术全解】08.Master主控节点服务-keepalived配置vip

admin2020年10月12日 09:33 Docker | Kubernetes | Linux 5人已围观

Docker&Kubernetes技术全解简介 Kubernetes 是一个可移植的、可扩展的开源平台,用于管理容器化的工作负载和服务,可促进声明式配置和自动化。Kubernetes 拥有一个庞大且快速增长的生态系统。Kubernetes 的服务、支持和工具广泛可用。 课程来自老男孩教育学习总结。

## 99.101/102 keepalived配置vip 配置keepalived实现nginx高可用,不能因为一个nginx宕机造成服务不可用。 ### 99.101配置keepalived主 #### 安装keepalived ```bash # 192.168.99.101 [root@k8s99-101 ~]# yum install keepalived -y ``` #### 监听端口脚本check_port.sh ```bash # 192.168.99.101 [root@k8s99-101 ~]# vim /etc/keepalived/check_port.sh # 写入以下内容 #!/bin/bash # keepalived监听nginx端口脚本 # 使用方法,在keepalived配置文件中 # vrrp_script chk_nginx { # script "/etc/keepalived/check_port.sh 7443" # .... CHK_PORT=$1 if [ -n "$CHK_PORT" ];then PORT_PROCESS=`ss -lnt | grep $CHK_PORT | wc -l` if [ $PORT_PROCESS -eq 0 ];then echo "Port $CHK_PORT is not used, please check." exit 1 fi else echo "Port $CHK_PORT is used, service is normal." fi ``` 这个脚本表示,如果nginx的7443端口没被监听了,表示nginx服务挂了,退出代码为1。 添加执行权限 ```bash # 192.168.99.101 [root@k8s99-101 ~]# chmod +x /etc/keepalived/check_port.sh # 测试这个脚本能否正常使用 [root@k8s99-101 ~]# /etc/keepalived/check_port.sh Port is used, service is normal. ``` #### 主keepalived配置 主和备的配置文件还是有所区别的 ```bash # 192.168.99.101 [root@k8s99-101 ~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak [root@k8s99-101 ~]# vim /etc/keepalived/keepalived.conf # 添加下面的内容到conf文件中 global_defs { # (VI_1) the virtual router id must be set,运行keepalived的机器的一个标识,通常可设为hostname。故障发生时,发邮件时显示在邮件主题中的信息 router_id chk_port_in_101 } vrrp_script chk_nginx { # 指定监控脚本,检测nginx服务是否正常运行 script "/etc/keepalived/check_port.sh 7443" # 指定监控时间,每2s执行一次 interval 2 # 脚本结果导致的优先级变更,检测失败(脚本返回非0)则优先级 -20 weight -20 } # keepalived在同一virtual_router_id中priority(0-255)最大的会成为master,也就是接管VIP,当priority最大的主机发生故障后,priority较小将会接管。 vrrp_instance VI_1 { # 指定keepalived的角色,主机设置为MASTER,备用机设置为BACKUP,注意这里的state指定instance(Initial)的初始状态, # 就是说在配置好后,这台服务器的初始状态就是这里指定的,但这里指定的不算,还是得要通过竞选通过优先级来确定。 # 如果这里设置为MASTER,但如若他的优先级不及另外一台,那么这台在发送通告时,会发送自己的优先级,另外一台发现优先级不如自己的高,那么他会就回抢占为MASTER。 state MASTER # 指定HA监测网络的接口。centos7使用 ip addr 获取 interface ens33 # 发送多播数据包时的源IP地址,这里实际上就是在哪个地址上发送VRRP通告,这个非常重要,一定要选择稳定的网卡端口来发送,这里相当于heartbeat的心跳端口, # 如果没有设置那么就用默认的绑定的网卡的IP,也就是interface指定的IP地址 mcast_src_ip 192.168.99.101 # 主备的virtual_router_id必须一样,可以设置为IP后一组:must be between 1 & 255 virtual_router_id 101 # 优先级值,在同一个vrrp_instance下, MASTRE 一定要高于 BAUCKUP,MASTER恢复后,BACKUP自动交接 priority 100 # VRRP 广播周期秒数,如果没检测到该广播,就被认为服务挂了,将切换主备 advert_int 1 # 设置为不抢占。默认是抢占的,当高优先级的机器恢复后,会抢占低优先级的机器成为MASTER,而不抢占,则允许低优先级的机器继续成为MASTER, # 即使高优先级的机器已经上线。如果要使用这个功能,则初始化状态必须为BACKUP。 nopreempt # 设置验证类型和密码。主从必须一样 authentication { # 设置vrrp验证类型,主要有PASS和AH两种 auth_type PASS # 加密的密码,两台服务器一定要一样,才能正常通信 auth_pass passwd } track_script { # 执行监控的服务,引用VRRP脚本,即在 vrrp_script 部分指定的名字。定期运行它们来改变优先级 chk_nginx } virtual_ipaddress { # VRRP HA 虚拟地址 如果有多个VIP,继续换行填写 192.168.99.100 } } ``` 注意: `state`主设置为`MASTER`,备设置为`BACKUP` `priority`主的只一定要大于备 `nopreempt`设置为不抢占模式 #### 防火墙允许VRRP路由冗余协议 否则主机之前keepalived无法正常通信 ```baswh [root@k8s99-101 ~]# firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT success [root@k8s99-101 ~]# firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT success [root@k8s99-101 ~]# firewall-cmd --reload success ``` 允许防火墙之后马上就在messages日志中看到keepalived的通信日志了,否则他们之前无法正常通信,vip在各自的主机上都会存在。 #### 启动keepalived ```bash [root@k8s99-101 ~]# systemctl start keepalived [root@k8s99-101 ~]# systemctl status keepalived [root@k8s99-101 ~]# systemctl enable keepalived ``` #### 检查vip所在的网卡 ```bash [root@k8s99-101 ~]# ip a show ens33 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:18:a0:76 brd ff:ff:ff:ff:ff:ff inet 192.168.99.101/24 brd 192.168.99.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.99.100/32 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::b3ad:c1c9:36cb:8198/64 scope link noprefixroute valid_lft forever preferred_lft forever ``` 可以通过`tail -f /var/log/messages`查到keepalived的日志。 ### 99.102配置keepalived备 ```bash # 192.168.99.102 # 安装keepalived yum install keepalived -y # 编写检测脚本 vim /etc/keepalived/check_port.sh #!/bin/bash # keepalived监听nginx端口脚本 # 使用方法,在keepalived配置文件中 # vrrp_script chk_nginx { # script "/etc/keepalived/check_port.sh 7443" # .... CHK_PORT=$1 if [ -n "$CHK_PORT" ];then PORT_PROCESS=`ss -lnt | grep $CHK_PORT | wc -l` if [ $PORT_PROCESS -eq 0 ];then echo "Port $CHK_PORT is not used, please check." exit 1 fi else echo "Port $CHK_PORT is used, service is normal." fi # 添加执行权限并尝试执行 chmod +x /etc/keepalived/check_port.sh /etc/keepalived/check_port.sh # 备份原来的keepalived配置 mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak # 创建新的keepalived配置 cat > /etc/keepalived/keepalived.conf << EOF global_defs { router_id chk_port_in_101 } vrrp_script chk_nginx { script "/etc/keepalived/check_port.sh 7443" interval 2 weight -20 } vrrp_instance VI_1 { state BACKUP interface ens33 mcast_src_ip 192.168.99.102 virtual_router_id 101 # 权重值 MASTRE 一定要高于 BAUCKUP priority 90 advert_int 1 authentication { auth_type PASS auth_pass passwd } track_script { chk_nginx } virtual_ipaddress { # VIP 虚拟IP地址 192.168.99.100 } } EOF # CentOS防火墙允许keepalived通信 firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT firewall-cmd --reload # 启动keepalived systemctl start keepalived systemctl status keepalived systemctl enable keepalived # 查看网卡信息,这是BACKUP,只要MASTER正常,vip不会漂移过来(前提是主备的keepalived通信是正常的) ip a show ens33 | grep 99.100 # 查看keepalived日志 tail -f /var/log/messages ``` ### 检查测试keepalived功能 #### 正常运行,查看主的状态 ```bash # 192.168.99.101 # 查看nginx运行状态 [root@k8s99-101 ~]# netstat -luntp | grep 7443 # 查看vip状态 [root@k8s99-101 ~]# ip a | grep 99.100 inet 192.168.99.100/32 scope global ens33 ``` 两者都正常运行时,vip在MASTER上 #### 正常运行,查看备的状态 ```bash # 192.168.99.102 [root@k8s99-102 ~]# netstat -luntp | grep 7443 tcp 0 0 0.0.0.0:7443 0.0.0.0:* LISTEN 1099/nginx: master [root@k8s99-102 ~]# ip a | grep 99.115 ``` #### 模拟nginx宕机,查看主的状态 ```bash # 192.168.99.101 [root@k8s99-101 ~]# nginx -s stop # vip已经漂移走了 [root@k8s99-101 ~]# ip a | grep 99.100 [root@k8s99-101 ~]# netstat -luntp | grep 7443 # 查看主的日志 [root@k8s99-101 ~]# tail -f /var/log/messages Jun 1 23:36:46 k8s99-101 Keepalived_vrrp[3572]: /etc/keepalived/check_port.sh 7443 exited with status 1 Jun 1 23:36:46 k8s99-101 Keepalived_vrrp[3572]: VRRP_Script(chk_nginx) failed Jun 1 23:36:46 k8s99-101 Keepalived_vrrp[3572]: VRRP_Instance(VI_1) Changing effective priority from 100 to 80 Jun 1 23:36:47 k8s99-101 Keepalived_vrrp[3572]: VRRP_Instance(VI_1) Received advert with higher priority 90, ours 80 Jun 1 23:36:47 k8s99-101 Keepalived_vrrp[3572]: VRRP_Instance(VI_1) Entering BACKUP STATE Jun 1 23:36:47 k8s99-101 Keepalived_vrrp[3572]: VRRP_Instance(VI_1) removing protocol VIPs. Jun 1 23:36:48 k8s99-101 Keepalived_vrrp[3572]: /etc/keepalived/check_port.sh 7443 exited with status 1 Jun 1 23:36:50 k8s99-101 Keepalived_vrrp[3572]: /etc/keepalived/check_port.sh 7443 exited with status 1 ``` #### 模拟nginx宕机,查看备的状态 ```bash # 192.168.99.102 # vip漂移到备上 [root@k8s99-102 ~]# ip a | grep 99.100 inet 192.168.99.100/32 scope global ens33 [root@k8s99-102 ~]# tail -f /var/log/messages Jun 1 23:35:12 k8s99-102 systemd: Reloading. Jun 1 23:36:47 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) forcing a new MASTER election Jun 1 23:36:48 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) Transition to MASTER STATE Jun 1 23:36:49 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) Entering MASTER STATE Jun 1 23:36:49 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) setting protocol VIPs. Jun 1 23:36:49 k8s99-102 Keepalived_vrrp[2038]: Sending gratuitous ARP on ens33 for 192.168.99.100 Jun 1 23:36:49 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.99.100 ``` 可以看到主k8s99-101上的vip就没有了,漂移到k8s99-102上面 #### 模拟主恢复,查看主的状态 当再次启动k8s99-101的nginx ```bash # 192.168.99.101 [root@k8s99-101 ~]# nginx [root@k8s99-101 ~]# netstat -luntp | grep 7443 tcp 0 0 0.0.0.0:7443 0.0.0.0:* LISTEN 16121/nginx: master # vip仍然没有漂移回来,因为配置了 [root@k8s99-101 ~]# ip a show ens33 | grep 99.100 # 查看日志 [root@k8s99-102 ~]# tail -f /var/log/messages Apr 25 22:56:14 k8s99-101 Keepalived_vrrp[14189]: /etc/keepalived/check_port.sh 7443 exited with status 1 Apr 25 22:56:16 k8s99-101 Keepalived_vrrp[14189]: VRRP_Script(chk_nginx) succeeded Apr 25 22:56:16 k8s99-101 Keepalived_vrrp[14189]: VRRP_Instance(VI_1) Changing effective priority from 80 to 100 ``` 但vip仍然没有回来,因为在主keepalived配置`vrrp_instance VI_1`中有一个`nopreempt`配置,即为非抢占式,可能会因为一些网络波动造成keepalived判断错误,vip漂移了。在**生产环境中vip不能随意动**的,就算主服务恢复正常后,vip也不能随意漂移回来。如果要让vip重新回来,需要**人工确认**好本机的服务完全正常才行。备上vip以及日志均无变化。 #### 主人工确认恢复,重启主的keepalived 在k8s99-101上进行人工确认 ```bash # 192.168.99.101 [root@k8s99-101 ~]# netstat -luntp | grep 7443 tcp 0 0 0.0.0.0:7443 0.0.0.0:* LISTEN 16121/nginx: master # 确实起来了,然后再重启keepalived [root@k8s99-101 ~]# systemctl restart keepalived # 查看日志 [root@k8s99-101 ~]# tail -f /var/log/messages Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: (VI_1): Warning - nopreempt will not work with initial state MASTER Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: SECURITY VIOLATION - scripts are being executed but script_security not enabled. Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: VRRP_Instance(VI_1) removing protocol VIPs. Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: Using LinkWatch kernel netlink reflector... Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)] Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: VRRP_Script(chk_nginx) succeeded Jun 1 23:42:32 k8s99-101 Keepalived_vrrp[5661]: VRRP_Instance(VI_1) Transition to MASTER STATE Jun 1 23:42:33 k8s99-101 Keepalived_vrrp[5661]: VRRP_Instance(VI_1) Entering MASTER STATE Jun 1 23:42:33 k8s99-101 Keepalived_vrrp[5661]: VRRP_Instance(VI_1) setting protocol VIPs. Jun 1 23:42:33 k8s99-101 Keepalived_vrrp[5661]: Sending gratuitous ARP on ens33 for 192.168.99.100 Jun 1 23:42:33 k8s99-101 Keepalived_vrrp[5661]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.99.100 # vip就回来了 [root@k8s99-101 ~]# ip a show ens33 | grep 99.100 inet 192.168.99.100/32 scope global ens33 ``` #### 查看备状态 再查看备的状态 ```bash # 192.168.99.102 [root@k8s99-102 ~]# tail -f /var/log/messages Jun 1 23:42:32 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) Received advert with higher priority 100, ours 90 Jun 1 23:42:32 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) Entering BACKUP STATE Jun 1 23:42:32 k8s99-102 Keepalived_vrrp[2038]: VRRP_Instance(VI_1) removing protocol VIPs. [root@k8s99-102 ~]# ip a | grep 99.115 # vip漂移到主上了 ```

很赞哦! (0)

文章交流

  • emoji
0人参与,0条评论

当前用户

未登录,点击   登录

站点信息

  • 建站时间:网站已运行792天
  • 系统信息:Linux
  • 后台程序:Python: 3.6.6
  • 网站框架:Django: 2.2.12
  • 文章统计:250 篇
  • 文章评论:45 条
  • 腾讯分析网站概况-腾讯分析
  • 百度统计网站概况-百度统计
  • 公众号:微信扫描二维码,关注我们
  • QQ群:QQ加群,下载网站的学习源码
返回
顶部
标题 换行 登录
网站