docker安装keepalived实现高可用
作者：互联网
一、认识keepalived
1、前言
nginx 作为负载均衡器，所有请求都到了 nginx，可见 nginx 处于非常重点的位置，如果nginx 服务器宕机后端 web 服务将无法提供服务，影响严重。
为了屏蔽负载均衡服务器的宕机，需要建立一个备份机。主服务器和备份机上都运行高可用（High Availability）监控程序（如keepalived），通过传送诸如“I am alive”这样的信息来监控对方的运行状况。当备份机不能在一定的时间内收到这样的信息时，它就接管主服务器的服务 IP 并继续提供负载均衡服务；当备份管理器又从主管理器收到“I am alive”这样的信息时，它就释放服务 IP 地址，这样的主服务器就开始再次提供负载均衡服务。
2、keepalived简介
keepalived 是集群管理中保证集群（如今天将的nginx集群）高可用的一个服务软件，用来防止单点故障。
Keepalived 的作用是检测 web 服务器（如nginx）的状态，如果有一台 web 服务器死机，或工作出现故障，（备机中的）Keepalived 将检测到，并将有故障的 web 服务器从系统中剔除，当 web 服务器工作正常后 Keepalived 自动将 web 服务器加入到服务器群中，这些工作全部自动完成，不需要人工干涉，需要人工做的只是修复故障的 web 服务器。
keepalived 是以 VRRP 协议为实现基础的，VRRP 全称 Virtual Router Redundancy Protocol，即虚拟路由冗余协议。
虚拟路由冗余协议，可以认为是实现路由器高可用的协议，即将 N 台提供相同功能的路由器组成一个路由器组，这个组里面有一个 master 和多个 backup，master 上面有一个对外提供服务的 vip（VIP = Virtual IPAddress，虚拟 IP 地址，该路由器所在局域网内其他机器的默认路由为该 vip），master 会发组播，当 backup 收不到 VRRP 包时就认为 master 宕掉了，这时就需要根据 VRRP 的优先级来选举一个 backup 当 master。这样的话就可以保证路由器的高可用了。
keepalived 主要有三个模块，分别是 core、check 和 VRRP。core 模块为 keepalived 的核心，负责主进程的启动、维护以及全局配置文件的加载和解析。check 负责健康检查，包括常见的各种检查方式。VRRP 模块是来实现 VRRP 协议的。
初始状态：
后端服务器集群（如nginx集群）通过VIP 193.168.101.100对外提供服务，客户端只知道VIP，并不关注后端服务器的真实地址。
主机宕机：
主机恢复：
virtual_ipaddress用于设置虚拟IP地址（VIP），又叫做漂移IP地址。可以设置多个虚拟IP地址，每行一个。之所以称为漂移IP地址，是因为Keepalived切换到Master状态时，这个IP地址会自动添加到（主服务器 的）系统中，而切换到BACKUP状态时，这些IP又会自动从（主服务器的）系统中删除。Keepalived通过“ip address add”命令的形式将VIP添加进系统中。要查看系统中添加的VIP地址，可以通过“ip add”命令实现。
“virtual_ipaddress”段中添加的IP形式可以多种多样，例如可以写成 “192.168.16.189/24 dev eth1” 这样的形式，而Keepalived会使用IP命令“ip addr add 192.168.16.189/24 dev eth1”将IP信息添加到系统中。因此，这里的配置规则和IP命令的使用规则是一致的。
虚拟IP(Virtual IP Address，简称VIP)是一个未分配给真实弹性云服务器网卡的IP地址。弹性云服务器除了拥有私有IP地址外，还可以拥有虚拟IP地址，用户可以通过其中任意一个IP(私有IP/虚拟IP)访问此弹性云服务器。同时，虚拟IP地址拥有私有IP地址同样的网络接入能力，包括VPC内二三层通信、VPC之间对等连接访问。虚拟IP地址用于为网卡提供第二个IP地址，同时支持与多个云服务器的网卡绑定，从而实现多个云服务器之间的高可用性。
VIP用于向客户端提供一个固定的“虚拟”访问地址，以避免后端服务器发生切换时对客户端的影响。VIP被加载在Master的网卡上，所有指向VIP的请求会被发向Master，Slave服务器出于Standby状态。如果Master出现故障，集群会通过选举算法从可用的Slave节点中选出一个新的Master节点，并将VIP也迁移到新Master节点的网卡上。这样可以保证服务始终可用，并且对客户端来说访问的IP也不会变化。注意VIP始终指向一个Master，因此VIP的方案并不能实现LB，只能实现HA。
二、我的华为云服务器配置虚拟IP
由于我的是华为云服务器，故只讲华为云服务器如何设置虚拟IP。
登录管理控制台，选择“总览 >云耀服务器”，在云耀服务器列表中，单击云耀服务器名称，系统跳转至该弹性云服务器详情页面，选择“网卡”页签，单击“管理虚拟IP地址"，进入如下页面：
点击申请虚拟IP地址
点击确定
点击绑定服务器
点击确定。
三、docker安装nginx和keepalived
下面我们来实现keepalived高可用，在centos7容器中安装nginx和keepalived。下图仅作示意，ip地址与本文无关。
1、运行两个centos容器
（1）、下载centos:7.6.1810镜像
docker pull centos:7.6.1810
定义自定义网络
docker network create --driver bridge --subnet 192.168.0.0/16 --gateway 192.168.0.1 mynet
（2）、运行两个centos容器
由于VIP要与keepalived的IP网段一致，故指定mynet网络。
docker run --privileged --cap-add SYS_ADMIN -e container=docker --name centos_master --net mynet -d centos:7.6.1810 /usr/sbin/init docker run --privileged --cap-add SYS_ADMIN -e container=docker --name centos_slave --net mynet -d centos:7.6.1810 /usr/sbin/init
这个命令用来建立一个CENTOS的容器。
--privileged 指定容器是否是特权容器。这里开启特权模式。
--cap-add SYS_ADMIN 添加系统的权限。不然，系统很多功能都用不了的。
-e container=docker 设置容器的类型。
/usr/sbin/init 初始容器里的CENTOS。
以上的参数是必需的。不然，建立的CENTOS容器不能正常使用和互动。
如果没有初始化和特权等等的开关，就不能使用systemctl。所以，以上的开关和设置是一样不能少的。
查看mynet网络：docker network inspect mynet
"Containers": {
"a7c5ac274402c817af76b0c32cded901a0625755c957a4ed8487a81aa9aa172a": {
"Name": "centos_slave",
"EndpointID": "c677d37936bd0d7c107d5d5f7a074cebd90900873df531f7eaaa8c8f3d1a85a9",
"MacAddress": "02:42:c0:a8:00:03",
"IPv4Address": "192.168.0.3/16",
"IPv6Address": ""
},
"c6e689818ca21969265e620261f0b0e07f19bc524847adaee170ff03e63b469c": {
"Name": "centos_master",
"EndpointID": "5848c8cfa67cef6ed143465b7a34b68660453a1cac696ed40e4697d3c0b5f69f",
"MacAddress": "02:42:c0:a8:00:02",
"IPv4Address": "192.168.0.2/16",
"IPv6Address": ""
}
},
由于要连接外网进行下载，故将my_centos_master和my_centos_slave加入到bridge。
docker network connect bridge centos_master docker network connect bridge centos_slave
查看bridge网络：docker network inspect bridge
"Containers": {
"a7c5ac274402c817af76b0c32cded901a0625755c957a4ed8487a81aa9aa172a": {
"Name": "centos_slave",
"EndpointID": "e44bcad2341c1fce318519a117fa5cd8e40b00a1d5f3db2911922c6760e729c6",
"MacAddress": "02:42:ac:11:00:03",
"IPv4Address": "172.17.0.3/16",
"IPv6Address": ""
},
"c6e689818ca21969265e620261f0b0e07f19bc524847adaee170ff03e63b469c": {
"Name": "centos_master",
"EndpointID": "c14aca283413d2d1b7e3cea7017dda2c9f5f7f11943469ef9652172220bd9b13",
"MacAddress": "02:42:ac:11:00:02",
"IPv4Address": "172.17.0.2/16",
"IPv6Address": ""
}
},
此时，centos_master和centos_slave既在bridge网络中，又在mynet网络中。
（3）、进入容器
查看容器：docker ps
[root@xxx ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a7c5ac274402 centos:7.6.1810 "/usr/sbin/init" 3 minutes ago Up 3 minutes centos_slave c6e689818ca2 centos:7.6.1810 "/usr/sbin/init" 3 minutes ago Up 3 minutes centos_master
进入容器
docker exec -it centos_master /bin/bash docker exec -it centos_slave /bin/bash
2、容器内安装Nginx的库和Nginx
（1）、先安装nginx的依赖库
rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
（2）、安装nginx
yum install -y nginx
查看nginx版本
[root@xxxx /]# nginx -v nginx version: nginx/1.22.0
说明nginx安装成功。
（3）、安装网络包(需要使用ifconfig和ping命令)
yum install -y iproute
yum install -y net-tools
输入ifconfig查看IP
主容器
[root@xxx /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.2 netmask 255.255.0.0 broadcast 192.168.255.255
ether 02:42:c0:a8:00:02 txqueuelen 0 (Ethernet)
RX packets 8 bytes 656 (656.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6 bytes 252 (252.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.2 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:ac:11:00:02 txqueuelen 0 (Ethernet)
RX packets 11489 bytes 29739901 (28.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11438 bytes 940859 (918.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 128 bytes 11208 (10.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 128 bytes 11208 (10.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
从容器
[root@a1ca509408ba /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.3 netmask 255.255.0.0 broadcast 192.168.255.255
ether 02:42:c0:a8:00:03 txqueuelen 0 (Ethernet)
RX packets 8 bytes 656 (656.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.3 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:ac:11:00:03 txqueuelen 0 (Ethernet)
RX packets 6860 bytes 29393745 (28.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6675 bytes 458937 (448.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 116 bytes 10414 (10.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 116 bytes 10414 (10.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
说明net-tools安装成功。
3、进入master 和slave容器修改标题
（1）、修改index.html文件
cd /usr/share/nginx/html vi index.html
主容器的nginx
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Master!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
从容器的nginx
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Slave!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
（2）、启动nginx
cd /usr/sbin/ ./nginx
知道如何启动nginx，同时也要知道如何关闭nginx
• 查看nginx进程id
ps -ef | grep nginx
• kill所有进程
killall nginx
（3）、检测master容器中nginx是否启动
curl localhost
结果：
[root@xxx sbin]# curl localhost
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Master!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
4、容器内安装keepalived
（1）、安装keepalived依赖环境
yum install -y gcc openssl-devel popt-devel
（2）、通过源码安装keepalived
yum -y install wget cd /root wget http://www.keepalived.org/software/keepalived-2.0.8.tar.gz --no-check-certificate tar -zxvf keepalived-2.0.8.tar.gz
rm -f keepalived-2.0.8.tar.gz cd keepalived-2.0.8 ./configure --prefix=/usr/local/keepalived make && make install
将keepalived.conf文件拷贝到/etc/keepalived
mkdir /etc/keepalived cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
（3）、在/etc/keepalived目录下创建Keepalived检测nginx的脚本check_nginx.sh，并赋权
cd /etc/keepalived touch check_nginx.sh vi check_nginx.sh
添加如下内容
A=`ps -ef | grep nginx | grep -v grep | wc -l`
if [ $A -eq 0 ];then
nginx
sleep 2
if [ `ps -ef | grep nginx | grep -v grep | wc -l` -eq 0 ];then
#killall keepalived
ps -ef|grep keepalived|grep -v grep|awk '{print $2}'|xargs kill -9
fi
fi
通过脚本检测nginx有没有挂，一旦挂了，就杀掉keepalived。这样备节点就成为了主节点。
（4）、给check_nginx.sh赋于执行权限
chmod +x /etc/keepalived/check_nginx.sh
（5）、编辑keepalived.conf
先编辑master的keepalived.conf文件
vi keepalived.conf
修改如下
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_MASTER
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight -20
fall 3
rise 2
user root
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 2
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.0.66
}
track_script {
chk_nginx
}
}
再编辑slave的keepalived.conf文件
vi keepalived.conf
修改如下
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_SLAVE
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight -20
fall 3
rise 2
user root
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 99
advert_int 2
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.0.66
}
track_script {
chk_nginx
}
}
5、容器内启动keepalived
（1）、启动keepalived
如果不能使用systemctl启动，可以按下面的方法启动
cd /usr/local/keepalived/sbin/ ./keepalived -f /etc/keepalived/keepalived.conf
关闭keepalived
ps -ef | grep keepalived kill -9 3747
（2）、使用ip add命令查看主备容器的ip情况
主容器
[root@a0d600c99ee7 sbin]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
146: eth0@if147: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet 192.168.0.66/32 scope global eth0
valid_lft forever preferred_lft forever
148: eth1@if149: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.2/16 brd 192.168.255.255 scope global eth1
valid_lft forever preferred_lft forever
从容器
[root@a1ca509408ba sbin]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
150: eth0@if151: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
152: eth1@if153: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.3/16 brd 192.168.255.255 scope global eth1
valid_lft forever preferred_lft forever
可以发现虚拟ip 192.168.0.66 此时已绑定在主容器上。
（5）、通过curl命令访问VIP
在master容器中访问192.168.0.66
如果报错如下：
[root@c6e689818ca2 sbin]# curl 192.168.0.66 curl: (7) Failed connect to 192.168.0.66:80; Connection timed out
修改keepalived.conf文件，将下面的内容注释掉即可。
vrrp_strict
keepalived默认使用组播，进行ARP广播，但是在云服务器上是禁止进行ARP广播的，容易造成ARP风暴。所以云服务器上部署或物理机部署采用单播的方式实现。
即：
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_MASTER
vrrp_skip_check_adv_addr
# vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight -20
fall 3
rise 2
user root
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 2
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.0.66
}
track_script {
chk_nginx
}
}
启动keepalived，再次通过curl命令访问192.168.0.66。
[root@a0d600c99ee7 sbin]# curl 192.168.0.66
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Master!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
在slave容器中访问192.168.0.66
[root@a0d600c99ee7 sbin]# curl 192.168.0.66
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Master!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
可以看现，此时master和slave容器两边通过虚拟vip : 192.168.0.66 访问nginx数据，请求返回的数据都是master容器中nginx配置的数据： welcome to nginx master。
继续验证，关掉master容器的keepalived服务：
ps -ef | grep keepalived kill -9 3747
注意：当备份机不能在一定的时间内收到“I am alive”这样的信息时，它就接管主服务器的服务 IP 并继续提供负载均衡服务；
查看主容器的IP
[root@a0d600c99ee7 sbin]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
146: eth0@if147: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
148: eth1@if149: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.2/16 brd 192.168.255.255 scope global eth1
valid_lft forever preferred_lft forever
查看从容器的IP
[root@a1ca509408ba sbin]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
150: eth0@if151: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet 192.168.0.66/32 scope global eth0
valid_lft forever preferred_lft forever
152: eth1@if153: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.3/16 brd 192.168.255.255 scope global eth1
valid_lft forever preferred_lft forever
可以发现虚拟ip 192.168.0.66 此时已绑定在从容器上。
此时在主容器访问curl 192.168.0.66
[root@a0d600c99ee7 sbin]# curl 192.168.0.66
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Slave!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
验证得到的结果是当master容器中的keepalived服务关掉后，curl 192.168.0.66请求返回的数据来自slave，welcome to nginx slave。在从容器中访问也是返回的welcome to nginx Slave。
再继续验证，把关掉master容器的keepalived服务再开启：
cd /usr/local/keepalived/sbin/ ./keepalived -f /etc/keepalived/keepalived.conf
注意：当备份管理器又从主管理器收到“I am alive”这样的信息时，它就释放服务 IP 地址，这样的主服务器就开始再次提供负载均衡服务。
此时查看主容器的IP
[root@a0d600c99ee7 sbin]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
146: eth0@if147: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet 192.168.0.66/32 scope global eth0
valid_lft forever preferred_lft forever
148: eth1@if149: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.2/16 brd 192.168.255.255 scope global eth1
valid_lft forever preferred_lft forever
发现虚拟ip 192.168.0.66 此时又绑定在主容器上。
此时再访问curl 192.168.0.66
[root@a0d600c99ee7 sbin]# curl 192.168.0.66
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Master!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
curl 192.168.0.66请求返回的数据来自Master，故可以说明已经实现了高可用。
四、先制作镜像再创建两个容器
根据第三条来创建两个容器，你会发现有大量的重复操作，我们可以先制作一个镜像，再根据该镜像创建两个容器，从而避免了大量重复操作。
1、配置模板容器
（1）、下载centos:7.6.1810镜像
docker pull centos:7.6.1810
（2）、启动并进入容器
docker run -it centos:7.6.1810 /bin/bash
（3）、安装基础工具
yum install -y iproute yum install -y net-tools
（4）、安装nginx的依赖库和安装nginx
先安装nginx依赖
rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
再安装nginx
yum install -y nginx
（5）、修改index.html的标题
cd /usr/share/nginx/html vi index.html
修改如下
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Master!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
（6）、启动nginx
cd /usr/sbin/ ./nginx
知道如何启动nginx，同时也要知道如何关闭nginx
• 查看nginx进程id
ps -ef | grep nginx
• kill所有进程
killall nginx
（7）、检测master容器中nginx是否启动
curl localhost
结果：
[root@07c519d6244b sbin]# curl localhost
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Master!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
（8）、容器内安装keepalived
安装keepalived依赖环境
yum install -y gcc openssl-devel popt-devel
通过源码安装keepalived
yum -y install wget cd /root wget http://www.keepalived.org/software/keepalived-2.0.8.tar.gz --no-check-certificate tar -zxvf keepalived-2.0.8.tar.gz
rm -f keepalived-2.0.8.tar.gz cd keepalived-2.0.8 ./configure --prefix=/usr/local/keepalived make && make install
将keepalived.conf文件拷贝到/etc/keepalived
mkdir /etc/keepalived cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
（9）、在/etc/keepalived目录下创建Keepalived检测nginx的脚本check_nginx.sh，并赋权
cd /etc/keepalived touch check_nginx.sh vi check_nginx.sh
添加如下内容
A=`ps -ef | grep nginx | grep -v grep | wc -l`
if [ $A -eq 0 ];then
nginx
sleep 2
if [ `ps -ef | grep nginx | grep -v grep | wc -l` -eq 0 ];then
#killall keepalived
ps -ef|grep keepalived|grep -v grep|awk '{print $2}'|xargs kill -9
fi
fi
通过脚本检测nginx有没有挂，一旦挂了，就杀掉keepalived。这样备节点就成为了主节点。
给check_nginx.sh赋于执行权限
chmod +x /etc/keepalived/check_nginx.sh
（10）、编辑keepalived.conf
先编辑master的keepalived.conf文件
vi keepalived.conf
修改如下
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_MASTER
vrrp_skip_check_adv_addr
# vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight -20
fall 3
rise 2
user root
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 2
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.0.66
}
track_script {
chk_nginx
}
}
（11）、退出容器，制作镜像
执行exit退出容器，查看容器
docker ps -a
结果
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 07c519d6244b centos:7.6.1810 "/bin/bash" 25 minutes ago Exited (1) 2 minutes ago peaceful_fermi
使用commit命令将容器保存为镜像
docker commit peaceful_fermi keepalived_nginx:v1
此镜像的内容就是当前容器的内容，接下来你可以用此镜像再次运行新的容器
2、启动主备keepalived容器
启动主容器
docker run -it \ --privileged=true \ --name keepalived_master \ --restart=always \ --net mynet --ip 192.168.0.2 \ keepalived_nginx:v1 \ /usr/sbin/init
去掉\改成一行执行
docker run -it --privileged=true --name keepalived_master --restart=always --net mynet --ip 192.168.0.2 keepalived_nginx:v1 /usr/sbin/init
启动从容器
docker run -it \ --privileged=true \ --name keepalived_slave \ --restart=always \ --net mynet --ip 192.168.0.3 \ keepalived_nginx:v1 \ /usr/sbin/init
去掉\改成一行执行
docker run -it --privileged=true --name keepalived_slave --restart=always --net mynet --ip 192.168.0.3 keepalived_nginx:v1 /usr/sbin/init
/usr/sbin/init 启动容器之后可以使用systemctl方法。
3、进入备机中修改keepalived配置
docker exec -it keepalived_slave bash cd /etc/keepalived vi keepalived.conf
修改如下：
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_SLAVE
vrrp_skip_check_adv_addr
# vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight -20
fall 3
rise 2
user root
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 99
advert_int 2
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.0.66
}
track_script {
chk_nginx
}
}
修改router_id、state、priority即可。
修改完后重启keepalived
cd /usr/local/keepalived/sbin/ ./keepalived -f /etc/keepalived/keepalived.conf
关闭keepalived
ps -ef | grep keepalived kill -9 3747
修改nginx中index.html的标题
cd /usr/share/nginx/html vi index.html
启动nginx
cd /usr/sbin/ ./nginx
4、测试高可用
进入主容器启动nginx和keepalived
查看主容器IP
[root@b3a23e52b268 sbin]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
188: eth0@if189: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.2/16 brd 192.168.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet 192.168.0.66/32 scope global eth0
valid_lft forever preferred_lft forever
进入从容器查看IP
[root@ad57dfb60f80 /]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
190: eth0@if191: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.3/16 brd 192.168.255.255 scope global eth0
valid_lft forever preferred_lft forever
主容器中通过curl命令访问192.168.0.66
[root@b3a23e52b268 sbin]# curl 192.168.0.66
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Master!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
从容器中通过curl命令访问192.168.0.66
[root@ad57dfb60f80 /]# curl 192.168.0.66
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Master!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
关闭主容器的keepalived
[root@b3a23e52b268 sbin]# ps -ef | grep keepalived root 3771 0 1 09:19 ? 00:00:04 ./keepalived -f /etc/keepalived/keepalived.conf root 3773 3771 0 09:19 ? 00:00:00 ./keepalived -f /etc/keepalived/keepalived.conf root 15815 3744 0 09:23 pts/1 00:00:00 grep --color=auto keepalived
杀死keepalived
kill -9 3771
在主容器中查看IP
[root@b3a23e52b268 sbin]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
188: eth0@if189: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.2/16 brd 192.168.255.255 scope global eth0
valid_lft forever preferred_lft forever
在从容器中查看IP
[root@ad57dfb60f80 /]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
190: eth0@if191: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.3/16 brd 192.168.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet 192.168.0.66/32 scope global eth0
valid_lft forever preferred_lft forever
发现VIP绑定到从容器中了。
在主容器中通过curl命令访问192.168.0.66
[root@b3a23e52b268 sbin]# curl 192.168.0.66
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Slave!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
在从容器中通过curl命令访问192.168.0.66
[root@ad57dfb60f80 sbin]# curl 192.168.0.66
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Slave!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
主容器中启动keepalived
主容器中查看ip
[root@b3a23e52b268 sbin]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
192: eth0@if193: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.2/16 brd 192.168.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet 192.168.0.66/32 scope global eth0
valid_lft forever preferred_lft forever
发现VIP又绑定到主容器上了。
在主容器中通过curl命令访问192.168.0.66
[root@b3a23e52b268 sbin]# curl 192.168.0.66
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx Master!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
说明keepalived实现了nginx的高可用。
标签：00,lft,keepalived,192.168,nginx,ff,docker,安装 来源： https://www.cnblogs.com/zwh0910/p/16518416.html