【K8s+Docker技术全解】05.部署k8s分布式数据库etcd

## 99.102/151/152部署etcd集群 ### 什么是etcd？ #### etcd简介 etcd是CoreOS团队于2013年6月发起的开源项目，它的目标是构建一个高可用的分布式键值(key-value)数据库。etcd内部采用`raft`协议作为一致性算法，etcd基于Go语言实现。 etcd作为服务发现系统，有以下的特点： - 简单：安装配置简单，而且提供了HTTP API进行交互，使用也很简单 - 安全：支持SSL证书验证 - 快速：根据官方提供的benchmark数据，单实例支持每秒2k+读操作 - 可靠：采用raft算法，实现分布式系统数据的可用性和一致性 是兼具一致性和高可用性的键值数据库，可以作为保存 Kubernetes 所有集群数据的后台数据库。在二进制部署etcd集群的时候，必须要考虑到高可用方案，一般部署三个或者三个以上的奇数个节点，因为当master宕机时，是通过选举制度来选择master的。 #### etcd应用场景 etcd比较多的应用场景是用于服务发现，服务发现(Service Discovery)要解决的是分布式系统中最常见的问题之一，即在同一个分布式集群中的进程或服务如何才能找到对方并建立连接。 从本质上说，服务发现就是要了解集群中是否有进程在监听upd或者tcp端口，并且通过名字就可以进行查找和连接。 要解决服务发现的问题，需要下面三大支柱，缺一不可。 - **一个强一致性、高可用的服务存储目录**。基于Ralf算法的etcd天生就是这样一个强一致性、高可用的服务存储目录。 - **一种注册服务和健康服务健康状况的机制**。用户可以在etcd中注册服务，并且对注册的服务配置key TTL，定时保持服务的心跳以达到监控健康状态的效果。 - **一种查找和连接服务的机制**。通过在etcd指定的主题下注册的服务也能在对应的主题下查找到。为了确保连接，我们可以在每个服务机器上都部署一个proxy模式的etcd，这样就可以确保访问etcd集群的服务都能够互相连接。 应用场景包括： - 配置管理 - 服务注册发现 - 选主 - 应用调度 - 分布式队列 - 分布式锁 `192.168.99.102`作为主etcd节点，`192.168.99.151`、`192.168.99.152`作为从etcd节点。 ### etcd指引（为 k8s 运行 etcd 集群） https://kubernetes.io/zh/docs/tasks/administer-cluster/configure-upgrade-etcd/ ### 运维主机签发etcd证书 #### 创建基于根证书的配置文件ca-config.json 安装etcd之前，要先给etcd签发证书，因为etcd之间的通信也是需要ssl的。 运维主机`192.168.99.200`创建生成证书的配置文件 ```bash # 192.168.99.200 [root@k8s99-200 ~]# cd /opt/certs/ [root@k8s99-200 certs]# vim ca-config.json ``` 证书配置如下 ```json { "signing": { "expiry": "175200h" }, "profiles": { "server": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth" ] }, "client": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "client auth" ] }, "peer": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } ``` `profiles`配置文件有三个段，分别表示证书类型： - `server`：服务端使用，客户端以此验证服务端身份，例如Docker服务端、kube-apiserver。 - `client`：客户端使用，用于服务端认证客户端，例如etcdctl、etcd proxy、fleetctl、Docker客户端。 - `peer`：双向证书，用户etcd集群成员间通信，客户端到服务端，服务端到客户端之间的通信都需要证书。 #### 创建etcd证书请求配置文件etcd-peer-csr.json 创建etcd的证书请求文件 ```bash # 192.168.99.200 [root@k8s99-200 certs]# vim etcd-peer-csr.json ``` 内容如下 ```json { "CN": "k8s_etcd", "hosts": [ "192.168.99.101", "192.168.99.102", "192.168.99.103", "192.168.99.151", "192.168.99.152", "192.168.99.153" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "sichuan", "L": "chengdu", "O": "study", "OU": "ops" } ] } ``` 这里和CA证书请求文件类似，但多了`hosts`段，表示etcd**可能**部署的主机，都需要添加到`hosts`里面，不能用网段，只能用IP。否则etcd之间的通信就会失败。 #### 生成基于该配置文件的证书 指定根证书`-ca=ca.pem`，根证书私钥`-ca-key=ca-key.pem`，根证书配置文件`-config=ca-config.json`，etcd证书请求配置文件`-profile=peer etcd-peer-csr.json` ```bash # 192.168.99.200 [root@k8s99-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json | cfssljson -bare etcd-peer 2020/06/01 21:59:43 [INFO] generate received request 2020/06/01 21:59:43 [INFO] received CSR 2020/06/01 21:59:43 [INFO] generating key: rsa-2048 2020/06/01 21:59:43 [INFO] encoded CSR 2020/06/01 21:59:43 [INFO] signed certificate with serial number 498562069613016908872218405276770200774750255243 2020/06/01 21:59:43 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements"). # 查看生成的etcd-peer证书 [root@k8s99-200 certs]# ll | grep "etcd-peer" -rw-r--r--. 1 root root 1082 6月 1 21:59 etcd-peer.csr -rw-r--r--. 1 root root 431 6月 1 21:58 etcd-peer-csr.json -rw-------. 1 root root 1679 6月 1 21:59 etcd-peer-key.pem -rw-r--r--. 1 root root 1452 6月 1 21:59 etcd-peer.pem ``` ### 99.102主etcd节点 #### 创建etcd用户 给etcd创建一个用户 ```bash # 192.168.99.102 [root@k8s99-102 ~]# useradd -s /sbin/nologin -M etcd [root@k8s99-102 ~]# id etcd uid=1000(etcd) gid=1000(etcd) 组=1000(etcd) ``` `-M, --no-create-home`表示不创建用户的主目录 #### 下载解压准备etcd 官网 https://etcd.io/ 下载 https://github.com/etcd-io/etcd/releases 这使用了最新的版本 [etcd-v3.4.5-linux-amd64.tar.gz](https://github.com/etcd-io/etcd/releases/download/v3.4.5/etcd-v3.4.5-linux-amd64.tar.gz) （下载太慢）换了 [etcd-v3.3.19-linux-amd64.tar.gz](https://github.com/etcd-io/etcd/releases/download/v3.3.19/etcd-v3.3.19-linux-amd64.tar.gz) 下载好就上传到服务器 ```bash # 192.168.99.102 [root@k8s99-102 ~]# ls etcd-v3.3.19-linux-amd64.tar.gz # 放在/opt目录，！！！如果直接在/root/下使用etcd，supervisor不能执行 [root@k8s99-102 ~]# mkdir /opt # 解压到/opt目录， [root@k8s99-102 ~]# tar zxf etcd-v3.3.19-linux-amd64.tar.gz -C /opt [root@k8s99-102 ~]# cd /opt/ [root@k8s99-102 opt]# ls etcd-v3.3.19-linux-amd64 # 可以只保留版本号 [root@k8s99-102 opt]# mv etcd-v3.3.19-linux-amd64 etcd-v3.3.19 # 创建软链接，直接使用etcd来访问，方便未来升级，直接修改软链接即可 [root@k8s99-102 opt]# ln -s etcd-v3.3.19 etcd [root@k8s99-102 opt]# ll 总用量 0 lrwxrwxrwx. 1 root root 12 6月 1 22:34 etcd -> etcd-v3.3.19 drwxr-xr-x. 3 630384594 600260513 123 3月 19 09:15 etcd-v3.3.19 # 查看etcd目录 [root@k8s99-102 opt]# ls etcd Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md ``` 在etcd目录里面比较重要的是`etcd`和`etcdctl`这两个脚本文件。 #### 创建etcd运行需要文件 ##### 创建etcd的log目录 创建etcd日志目录 ```bash # 192.168.99.102 [root@k8s99-102 opt]# mkdir /var/log/etcd-server # 修改属主和属组 [root@k8s99-102 opt]# chown -R etcd.etcd /var/log/etcd-server/ [root@k8s99-102 opt]# ls -ld /var/log/etcd-server drwxr-xr-x. 2 etcd etcd 6 3月 21 22:07 /var/log/etcd-server ``` 即后面配置etcd时将它的日志存放在`/var/log/etcd-server`目录 ##### 创建etcd的data目录 在etcd目录下创建`data`目录，即`/opt/etcd/data`，也可以创建在其他地方，用于存放etcd相关的数据文件 ```bash # 192.168.99.102 [root@k8s99-102 opt]# cd etcd [root@k8s99-102 etcd]# mkdir data ``` ##### 从运维主机拷贝证书、私钥到certs 将运维管理主机`192.168.99.200`上生成的根证书文件`ca.pem`、etcd证书文件`etcd-peer.pem`、etcd证书私钥`etcd-peer-key.pem`拷贝到刚创建的`/root/etcd/certs`目录。 在etcd目录中创建`certs`目录，用于存放证书和私钥 ```bash # 192.168.99.102 [root@k8s99-102 etcd]# mkdir certs [root@k8s99-102 etcd]# cd certs/ [root@k8s99-102 certs]# pwd /opt/etcd/certs ``` 开始拷贝证书文件 ```bash # 192.168.99.102 [root@k8s99-102 etcd]# cd certs/ # 从运维主机复制根证书文件 [root@k8s99-102 certs]# scp k8s99-200:/opt/certs/ca.pem . The authenticity of host 'k8s99-200 (192.168.99.200)' can't be established. ECDSA key fingerprint is SHA256:sZ8YJcYAarwkAZg1GiHrQJpVRdzLBLtTma6o8Q8nSt4. ECDSA key fingerprint is MD5:6f:83:14:23:29:e2:3e:33:0a:a1:69:cd:dc:63:5b:df. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'k8s99-200,192.168.99.200' (ECDSA) to the list of known hosts. root@k8s99-200's password: ca.pem 100% 1354 1.3MB/s 00:00 # 复制etcd证书文件 [root@k8s99-102 certs]# scp k8s99-200:/opt/certs/etcd-peer.pem . root@k8s99-200`s password: etcd-peer.pem 100% 1436 1.1MB/s 00:00 # 复制etcd证书私钥文件 [root@k8s99-102 certs]# scp k8s99-200:/opt/certs/etcd-peer-key.pem . root@k8s99-200`s password: etcd-peer-key.pem 100% 1679 1.1MB/s 00:00 # 若果不想每次都输入密码，执行： echo 密码 | passwd root --stdin ``` 证书私钥key可以看到**权限为600，在生产环境中是不能给他人的**。 #### etcd命令帮助 ```bash # 192.168.99.102 [root@k8s99-102 certs]# cd .. [root@k8s99-102 etcd]# ./etcd --help usage: etcd [flags] start an etcd server etcd --version show the version of etcd etcd -h | --help show the help information about etcd etcd --config-file path to the server configuration file etcd gateway run the stateless pass-through etcd TCP connection forwarding proxy etcd grpc-proxy run the stateless etcd v3 gRPC L7 reverse proxy member flags: --name 'default' human-readable name for this member. --data-dir '${name}.etcd' path to the data directory. --wal-dir '' path to the dedicated wal directory. --snapshot-count '100000' number of committed transactions to trigger a snapshot to disk. --heartbeat-interval '100' time (in milliseconds) of a heartbeat interval. --election-timeout '1000' time (in milliseconds) for an election to timeout. See tuning documentation for details. --initial-election-tick-advance 'true' whether to fast-forward initial election ticks on boot for faster election. --listen-peer-urls 'http://localhost:2380' list of URLs to listen on for peer traffic. --listen-client-urls 'http://localhost:2379' list of URLs to listen on for client traffic. --max-snapshots '5' maximum number of snapshot files to retain (0 is unlimited). --max-wals '5' maximum number of wal files to retain (0 is unlimited). --cors '' comma-separated whitelist of origins for CORS (cross-origin resource sharing). --quota-backend-bytes '0' raise alarms when backend size exceeds the given quota (0 defaults to low space quota). --max-txn-ops '128' maximum number of operations permitted in a transaction. --max-request-bytes '1572864' maximum client request size in bytes the server will accept. --grpc-keepalive-min-time '5s' minimum duration interval that a client should wait before pinging server. --grpc-keepalive-interval '2h' frequency duration of server-to-client ping to check if a connection is alive (0 to disable). --grpc-keepalive-timeout '20s' additional duration of wait before closing a non-responsive connection (0 to disable). clustering flags: --initial-advertise-peer-urls 'http://localhost:2380' list of this member's peer URLs to advertise to the rest of the cluster. --initial-cluster 'default=http://localhost:2380' initial cluster configuration for bootstrapping. --initial-cluster-state 'new' initial cluster state ('new' or 'existing'). --initial-cluster-token 'etcd-cluster' initial cluster token for the etcd cluster during bootstrap. Specifying this can protect you from unintended cross-cluster interaction when running multiple clusters. --advertise-client-urls 'http://localhost:2379' list of this member's client URLs to advertise to the public. The client URLs advertised should be accessible to machines that talk to etcd cluster. etcd client libraries parse these URLs to connect to the cluster. --discovery '' discovery URL used to bootstrap the cluster. --discovery-fallback 'proxy' expected behavior ('exit' or 'proxy') when discovery services fails. "proxy" supports v2 API only. --discovery-proxy '' HTTP proxy to use for traffic to discovery service. --discovery-srv '' dns srv domain used to bootstrap the cluster. --strict-reconfig-check 'true' reject reconfiguration requests that would cause quorum loss. --auto-compaction-retention '0' auto compaction retention length. 0 means disable auto compaction. --auto-compaction-mode 'periodic' interpret 'auto-compaction-retention' one of: periodic|revision. 'periodic' for duration based retention, defaulting to hours if no time unit is provided (e.g. '5m'). 'revision' for revision number based retention. --enable-v2 'true' Accept etcd V2 client requests. proxy flags: "proxy" supports v2 API only. --proxy 'off' proxy mode setting ('off', 'readonly' or 'on'). --proxy-failure-wait 5000 time (in milliseconds) an endpoint will be held in a failed state. --proxy-refresh-interval 30000 time (in milliseconds) of the endpoints refresh interval. --proxy-dial-timeout 1000 time (in milliseconds) for a dial to timeout. --proxy-write-timeout 5000 time (in milliseconds) for a write to timeout. --proxy-read-timeout 0 time (in milliseconds) for a read to timeout. security flags: --ca-file '' [DEPRECATED] path to the client server TLS CA file. '-ca-file ca.crt' could be replaced by '-trusted-ca-file ca.crt -client-cert-auth' and etcd will perform the same. --cert-file '' path to the client server TLS cert file. --key-file '' path to the client server TLS key file. --client-cert-auth 'false' enable client cert authentication. --client-crl-file '' path to the client certificate revocation list file. --trusted-ca-file '' path to the client server TLS trusted CA cert file. --auto-tls 'false' client TLS using generated certificates. --peer-ca-file '' [DEPRECATED] path to the peer server TLS CA file. '-peer-ca-file ca.crt' could be replaced by '-peer-trusted-ca-file ca.crt -peer-client-cert-auth' and etcd will perform the same. --peer-cert-file '' path to the peer server TLS cert file. --peer-key-file '' path to the peer server TLS key file. --peer-client-cert-auth 'false' enable peer client cert authentication. --peer-trusted-ca-file '' path to the peer server TLS trusted CA file. --peer-cert-allowed-cn '' Required CN for client certs connecting to the peer endpoint. --peer-auto-tls 'false' peer TLS using self-generated certificates if --peer-key-file and --peer-cert-file are not provided. --peer-crl-file '' path to the peer certificate revocation list file. --cipher-suites '' comma-separated list of supported TLS cipher suites between client/server and peers (empty will be auto-populated by Go). --experimental-peer-skip-client-san-verification 'false' Skip verification of SAN field in client certificate for peer connections. logging flags --debug 'false' enable debug-level logging for etcd. --log-package-levels '' specify a particular log level for each etcd package (eg: 'etcdmain=CRITICAL,etcdserver=DEBUG'). --log-output 'default' specify 'stdout' or 'stderr' to skip journald logging even when running under systemd. unsafe flags: Please be CAUTIOUS when using unsafe flags because it will break the guarantees given by the consensus protocol. --force-new-cluster 'false' force to create a new one-member cluster. profiling flags: --enable-pprof 'false' Enable runtime profiling data via HTTP server. Address is at client URL + "/debug/pprof/" --metrics 'basic' Set level of detail for exported metrics, specify 'extensive' to include histogram metrics. --listen-metrics-urls '' List of URLs to listen on for metrics. auth flags: --auth-token 'simple' Specify a v3 authentication token type and its options ('simple' or 'jwt'). experimental flags: --experimental-initial-corrupt-check 'false' enable to check data corruption before serving any client/peer traffic. --experimental-corrupt-check-time '0s' duration of time between cluster corruption check passes. --experimental-enable-v2v3 '' serve v2 requests through the v3 backend under a given prefix. ``` #### 创建etcd服务启动脚本 创建启动脚本`etcd-server-startup.sh` ```bash # 192.168.99.102 [root@k8s99-102 etcd]# vim etcd-server-startup.sh # 写入下面的脚本 #!/bin/bash ./etcd --name etcd-server-k8s99-102 \ --data-dir /opt/etcd/data \ --listen-peer-urls https://192.168.99.102:2380 \ --listen-client-urls https://192.168.99.102:2379,http://127.0.0.1:2379 \ --quota-backend-bytes 8000000000 \ --initial-advertise-peer-urls https://192.168.99.102:2380 \ --advertise-client-urls https://192.168.99.102:2379,http://127.0.0.1:2379 \ --initial-cluster etcd-server-k8s99-102=https://192.168.99.102:2380,etcd-server-k8s99-151=https://192.168.99.151:2380,etcd-server-k8s99-152=https://192.168.99.152:2380 \ --ca-file ./certs/ca.pem \ --cert-file ./certs/etcd-peer.pem \ --key-file ./certs/etcd-peer-key.pem \ --client-cert-auth \ --trusted-ca-file ./certs/ca.pem \ --peer-ca-file ./certs/ca.pem \ --peer-cert-file ./certs/etcd-peer.pem \ --peer-key-file ./certs/etcd-peer-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file ./certs/ca.pem \ --log-output stdout ``` 添加执行权限 ```bash # 192.168.99.102 [root@k8s99-102 etcd]# chmod +x etcd-server-startup.sh ``` 注意脚本每一行后面不要加注释 - `--name`：启动名称 - `--data-dir`：指定etcd存放数据的位置 - `listen-peer-urls`：内部服务使用2380端口 - `--listen-client-urls`：对外的客户端提供服务使用2379端口 - `--quota-backend-bytes`：后端配额 - `--initial-advertise-peer-urls`：初始通知对等地址 - `--advertise-client-urls`：通知客户端的地址 - `--initial-cluster`：初始化集群列表，内部服务使用2380端口 - `--client-cert-auth`：指定需要验证证书 这样就可以通过`./etcd-server-startup.sh`启动etcd服务了，让其正常运行一下，目的是为了生成etcd运行时的目录，例如`[root@k8s99-102 etcd]# ls data/`。但是为了让其能够在后端运行，需要安装supervisor #### 修改etcd运行目录所有者和所属组 将整个`/root/etcd/`目录修改为该属主和属组。后面使用supervisor运行时涉及到权限相关，非常重要 ```bash # 192.168.99.102 [root@k8s99-102 etcd]# chown -R etcd.etcd /opt/etcd/ # 确认etcd目录下的文件 [root@k8s99-102 etcd]# ll 总用量 39520 drwxr-xr-x. 2 etcd etcd 66 4月 22 22:36 certs drwxr-xr-x. 3 etcd etcd 20 4月 22 23:13 data drwxr-xr-x. 10 etcd etcd 4096 3月 19 09:15 Documentation -rwxr-xr-x. 1 etcd etcd 22389440 3月 19 09:15 etcd -rwxr-xr-x. 1 etcd etcd 18008256 3月 19 09:15 etcdctl -rwxr-xr-x. 1 etcd etcd 950 4月 22 22:50 etcd-server-startup.sh -rw-r--r--. 1 etcd etcd 38864 3月 19 09:15 README-etcdctl.md -rw-r--r--. 1 etcd etcd 7262 3月 19 09:15 README.md -rw-r--r--. 1 etcd etcd 7855 3月 19 09:15 READMEv2-etcdctl.md # 确认证书文件 [root@k8s99-102 etcd]# ll certs/ 总用量 12 -rw-r--r--. 1 etcd etcd 1354 4月 22 22:35 ca.pem -rw-------. 1 etcd etcd 1679 4月 22 22:36 etcd-peer-key.pem -rw-r--r--. 1 etcd etcd 1436 4月 22 22:35 etcd-peer.pem # 确认运行etcd生成的data文件 [root@k8s99-102 etcd]# ll data/ 总用量 0 drwx------. 4 etcd etcd 29 4月 22 23:13 member [root@k8s99-102 etcd]# ll data/member/ 总用量 0 drwx------. 2 etcd etcd 16 4月 22 23:13 snap drwx------. 2 etcd etcd 64 4月 22 23:13 wal [root@k8s99-102 etcd]# ll data/member/snap/ 总用量 20 -rw-------. 1 etcd etcd 20480 4月 22 23:13 db [root@k8s99-102 etcd]# ll data/member/wal/ ``` #### 使用supervisor运行etcd ##### 安装supervisor 管理后台进程，让etcd自动掉线运行。 ```bash # 192.168.99.102 # 安装 [root@k8s99-102 etcd]# yum install supervisor -y # 启动并设置开机自启 [root@k8s99-102 etcd]# systemctl start supervisord [root@k8s99-102 etcd]# systemctl status supervisord [root@k8s99-102 etcd]# systemctl enable supervisord ``` ##### 创建etcd的supervisor启动配置 创建etcd-server的supervisor配置文件 ```bash # 192.168.99.102 [root@k8s99-102 etcd]# vim /etc/supervisord.d/etcd-server.ini # 如以下内容 ``` ```ini [program:etcd-server-k8s99-102] command=/bin/bash /opt/etcd/etcd-server-startup.sh numprocs=1 directory=/opt/etcd autostart=true autorestart=true startsecs=30 startretries=3 exitcodes=0,2 stopsignal=QUIT stopwaitsecs=10 user=etcd redirect_stderr=true stdout_logfile=/var/log/etcd-server/etcd.supervisor.log stdout_logfile_maxbytes=64MB stdout_logfile_backups=4 stdout_capture_maxbytes=1MB stdout_events_enabled=false ``` ##### 启动etcd-server-k8s99-102 ```bash # 192.168.99.102 [root@k8s99-102 etcd]# supervisorctl update [root@k8s99-102 etcd]# supervisorctl status etcd-server-k8s99-102 FATAL Exited too quickly (process log may have details) [root@k8s99-102 etcd]# cat /var/log/etcd-server/etcd.supervisor.log # ========将etcd目录放在/root/目录下报错，所以调整到/opt/目录下========== supervisor: couldn`t chdir to /root/etcd: EACCES supervisor: child process was not spawned # ========将etcd目录放在/opt/下报错，command需要添加/bin/bash，或者sh文件中添加头：#!/bin/bash========== supervisor: couldn`t exec /opt/etcd/etcd-server-startup.sh: ENOEXEC supervisor: child process was not spawned # ========打开证书权限问题，需要将所有证书文件修改所有者和所属组======== 2020-04-22 22:51:05.261034 C | etcdmain: open ./certs/etcd-peer-key.pem: permission denied 2020-04-22 22:51:07.650077 C | etcdmain: open ./certs/etcd-peer-key.pem: permission denied 2020-04-22 22:51:10.792915 C | etcdmain: open ./certs/etcd-peer-key.pem: permission denied # ========在data下面创建目录无权限，确认data/的所有者和所属组======== 2020-04-22 23:09:47.606644 C | etcdserver: create snapshot directory error: mkdir /opt/etcd/data/xx***xx/xx***xx: permission denied ``` 如果出现上面的问题，检查完就进行下面的操作，直到状态为`STARTING`即可 ```bash # 如果使用supervisorctl update不能更新，重启下supervisord服务 [root@k8s99-102 etcd]# systemctl restart supervisord [root@k8s99-102 etcd]# supervisorctl status etcd-server-k8s99-102 STARTING # 然后变为RUNNING [root@k8s99-102 etcd]# supervisorctl status etcd-server-k8s99-102 RUNNING pid 1627, uptime 0:00:30 ``` #### 查看启动运行的端口 可以使用`tail -f /var/log/etcd-server/etcd.supervisor.log`查看etcd是否启动完成，然后再确认端口 ```bash # 192.168.99.102 [root@k8s99-102 etcd]# netstat -luntp | grep etcd tcp 0 0 192.168.99.102:2379 0.0.0.0:* LISTEN 1628/./etcd tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 1628/./etcd tcp 0 0 192.168.99.102:2380 0.0.0.0:* LISTEN 1628/./etcd ``` 可以看到etcd监听了2379和2380两个端口，表示启动成功了。 #### 防火墙允许这两个端口 ```bash # 192.168.99.102 [root@k8s99-102 etcd]# firewall-cmd --zone=public --add-port=2379-2380/tcp --permanent success [root@k8s99-102 etcd]# firewall-cmd --reload success ``` ### 99.151备etcd节点 ```bash # 192.168.99.151 # 创建etcd用户 useradd -s /sbin/nologin -M etcd # 进入用户目录 cd ~ # 创建etcd运行目录 mkdir /opt tar zxf etcd-v3.3.19-linux-amd64.tar.gz -C /opt # 准备etcd运行目录 cd /opt mv etcd-v3.3.19-linux-amd64 etcd-v3.3.19 ln -s etcd-v3.3.19 etcd ls etcd # 创建log目录 mkdir /var/log/etcd-server chown -R etcd.etcd /var/log/etcd-server/ ls -ld /var/log/etcd-server # 创建certs目录 cd etcd mkdir certs cd certs/ # ============手动 # 拷贝证书，输入密码，需要手动进行 scp k8s99-200:/opt/certs/ca.pem . scp k8s99-200:/opt/certs/etcd-peer.pem . scp k8s99-200:/opt/certs/etcd-peer-key.pem . cd .. # 创建data目录 mkdir data # 需要修改--name的值，--listen-peer-urls、--listen-client-urls、--initial-advertise-peer-urls、--advertise-client-urls改为本机的IP echo "./etcd --name etcd-server-k8s99-151 \ --data-dir /opt/etcd/data \ --listen-peer-urls https://192.168.99.151:2380 \ --listen-client-urls https://192.168.99.151:2379,http://127.0.0.1:2379 \ --quota-backend-bytes 8000000000 \ --initial-advertise-peer-urls https://192.168.99.151:2380 \ --advertise-client-urls https://192.168.99.151:2379,http://127.0.0.1:2379 \ --initial-cluster etcd-server-k8s99-102=https://192.168.99.102:2380,etcd-server-k8s99-151=https://192.168.99.151:2380,etcd-server-k8s99-152=https://192.168.99.152:2380 \ --ca-file ./certs/ca.pem \ --cert-file ./certs/etcd-peer.pem \ --key-file ./certs/etcd-peer-key.pem \ --client-cert-auth \ --trusted-ca-file ./certs/ca.pem \ --peer-ca-file ./certs/ca.pem \ --peer-cert-file ./certs/etcd-peer.pem \ --peer-key-file ./certs/etcd-peer-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file ./certs/ca.pem \ --log-output stdout" > etcd-server-startup.sh # 修改执行权限 chmod +x etcd-server-startup.sh # 尝试运行，这一步主要是为了在data下生成相关的文件 ./etcd-server-startup.sh # 修改所有者和所属组 chown -R etcd.etcd /opt/etcd/ ll ll certs/ ll data/ # 安装supervisor yum install supervisor -y systemctl start supervisord systemctl status supervisord systemctl enable supervisord # 修改启动配置，主要要修改名称 echo "[program:etcd-server-k8s99-151] command=/bin/bash /opt/etcd/etcd-server-startup.sh numprocs=1 directory=/opt/etcd autostart=true autorestart=true startsecs=30 startretries=3 exitcodes=0,2 stopsignal=QUIT stopwaitsecs=10 user=etcd redirect_stderr=true stdout_logfile=/var/log/etcd-server/etcd.supervisor.log stdout_logfile_maxbytes=64MB stdout_logfile_backups=4 stdout_capture_maxbytes=1MB stdout_events_enabled=false" > /etc/supervisord.d/etcd-server.ini supervisorctl update supervisorctl status # 查看启动的端口 netstat -luntp | grep etcd # 如果防火墙开启，就允许端口 firewall-cmd --zone=public --add-port=2379-2380/tcp --permanent firewall-cmd --reload # 查看etcd日志 tail -f /var/log/etcd-server/etcd.supervisor.log ``` ### 99.152备etcd节点 ```bash # 192.168.99.152 # 创建etcd用户 useradd -s /sbin/nologin -M etcd # 进入用户目录 cd ~ # 创建etcd运行目录 mkdir /opt tar zxf etcd-v3.3.19-linux-amd64.tar.gz -C /opt # 准备etcd运行目录 cd /opt mv etcd-v3.3.19-linux-amd64 etcd-v3.3.19 ln -s etcd-v3.3.19 etcd ls etcd # 创建log目录 mkdir /var/log/etcd-server chown -R etcd.etcd /var/log/etcd-server/ ls -ld /var/log/etcd-server # 创建certs目录 cd etcd mkdir certs cd certs/ # 拷贝证书，输入密码，需要手动进行 scp k8s99-200:/opt/certs/ca.pem . scp k8s99-200:/opt/certs/etcd-peer.pem . scp k8s99-200:/opt/certs/etcd-peer-key.pem . cd .. # 创建data目录 mkdir data # 需要修改--name的值，--listen-peer-urls、--listen-client-urls、--initial-advertise-peer-urls、--advertise-client-urls改为本机的IP echo "./etcd --name etcd-server-k8s99-152 \ --data-dir /opt/etcd/data \ --listen-peer-urls https://192.168.99.152:2380 \ --listen-client-urls https://192.168.99.152:2379,http://127.0.0.1:2379 \ --quota-backend-bytes 8000000000 \ --initial-advertise-peer-urls https://192.168.99.152:2380 \ --advertise-client-urls https://192.168.99.152:2379,http://127.0.0.1:2379 \ --initial-cluster etcd-server-k8s99-102=https://192.168.99.102:2380,etcd-server-k8s99-151=https://192.168.99.151:2380,etcd-server-k8s99-152=https://192.168.99.152:2380 \ --ca-file ./certs/ca.pem \ --cert-file ./certs/etcd-peer.pem \ --key-file ./certs/etcd-peer-key.pem \ --client-cert-auth \ --trusted-ca-file ./certs/ca.pem \ --peer-ca-file ./certs/ca.pem \ --peer-cert-file ./certs/etcd-peer.pem \ --peer-key-file ./certs/etcd-peer-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file ./certs/ca.pem \ --log-output stdout" > etcd-server-startup.sh # 修改执行权限 chmod +x etcd-server-startup.sh # 尝试运行，这一步主要是为了在data下生成相关的文件 ./etcd-server-startup.sh # 结束运行 # 修改所有者和所属组 chown -R etcd.etcd /opt/etcd/ ll ll certs/ ll data/ # 安装supervisor yum install supervisor -y systemctl start supervisord systemctl status supervisord systemctl enable supervisord # 修改启动配置，主要要修改名称 echo "[program:etcd-server-k8s99-152] command=/bin/bash /opt/etcd/etcd-server-startup.sh numprocs=1 directory=/opt/etcd autostart=true autorestart=true startsecs=30 startretries=3 exitcodes=0,2 stopsignal=QUIT stopwaitsecs=10 user=etcd redirect_stderr=true stdout_logfile=/var/log/etcd-server/etcd.supervisor.log stdout_logfile_maxbytes=64MB stdout_logfile_backups=4 stdout_capture_maxbytes=1MB stdout_events_enabled=false" > /etc/supervisord.d/etcd-server.ini supervisorctl update supervisorctl status # 查看启动的端口 netstat -luntp | grep etcd # 如果防火墙开启，就允许端口 firewall-cmd --zone=public --add-port=2379-2380/tcp --permanent firewall-cmd --reload # 查看etcd日志 tail -f /var/log/etcd-server/etcd.supervisor.log ``` ### 检测集群健康状态 ```bash # 192.168.99.102 [root@k8s99-102 ~]# tail -f /var/log/etcd-server/etcd.supervisor.log # ... 2020-06-01 22:50:58.799374 W | etcdserver: failed to reach the peerURL(https://192.168.99.152:2380) of member 1175991420b7dee6 (Get https://192.168.99.152:2380/version: dial tcp 192.168.99.152:2380: i/o timeout) 2020-06-01 22:50:58.799515 W | etcdserver: cannot get the version of member 1175991420b7dee6 (Get https://192.168.99.152:2380/version: dial tcp 192.168.99.152:2380: i/o timeout) 2020-06-01 22:51:02.634006 W | rafthttp: health check for peer 1175991420b7dee6 could not connect: dial tcp 192.168.99.152:2380: connect: no route to host (prober "ROUND_TRIPPER_RAFT_MESSAGE") 2020-06-01 22:51:02.634035 W | rafthttp: health check for peer 1175991420b7dee6 could not connect: dial tcp 192.168.99.152:2380: connect: no route to host (prober "ROUND_TRIPPER_SNAPSHOT") 2020-06-01 22:51:03.668790 I | rafthttp: peer 1175991420b7dee6 became active 2020-06-01 22:51:03.701785 I | rafthttp: established a TCP streaming connection with peer 1175991420b7dee6 (stream MsgApp v2 reader) 2020-06-01 22:51:03.805455 I | rafthttp: established a TCP streaming connection with peer 1175991420b7dee6 (stream Message reader) 2020-06-01 22:51:03.813460 I | etcdserver: updating the cluster version from 3.0 to 3.3 2020-06-01 22:51:03.814735 N | etcdserver/membership: updated the cluster version from 3.0 to 3.3 2020-06-01 22:51:03.814782 I | etcdserver/api: enabled capabilities for version 3.3 ``` 在配置152节点前，102节点一直监视着日志，直到152配置完启动完成后，就出现了上面的日志了。 在任意一个节点可以使用`etcdctl`命令去检测集群的健康状态 ```bash # 192.168.99.102 [root@k8s99-102 etcd]# ls certs data Documentation etcd etcdctl etcd-server-startup.sh README-etcdctl.md README.md READMEv2-etcdctl.md # 检查集群健康状态 [root@k8s99-102 etcd]# ./etcdctl cluster-health member 1175991420b7dee6 is healthy: got healthy result from http://127.0.0.1:2379 member 363c8a8932abb3bb is healthy: got healthy result from http://127.0.0.1:2379 member 480be059079c091b is healthy: got healthy result from http://127.0.0.1:2379 cluster is healthy ``` 可以看到当个etcd节点启动后，每个节点都是正常的。 另外还有一种检查方法 ```bash # 192.168.99.102 [root@k8s99-102 etcd]# ./etcdctl member list 1175991420b7dee6: name=etcd-server-k8s99-152 peerURLs=https://192.168.99.152:2380 clientURLs=http://127.0.0.1:2379,https://192.168.99.152:2379 isLeader=false 363c8a8932abb3bb: name=etcd-server-k8s99-151 peerURLs=https://192.168.99.151:2380 clientURLs=http://127.0.0.1:2379,https://192.168.99.151:2379 isLeader=false 480be059079c091b: name=etcd-server-k8s99-102 peerURLs=https://192.168.99.102:2380 clientURLs=http://127.0.0.1:2379,https://192.168.99.102:2379 isLeader=true ``` 即可以看到最先启动好的是`isLeader=true`，即主节点，另外两个都是备节点。

明人不说暗话 领红包就是支持！！！ 来都来了，再到页面底部看下呗~ ↓ ↓ ↓

本文地址： http://blog.starmeow.cn/detail/7a614fd06c325499f1680b9896beedeb/

很赞哦！ (0)