【kubernetes】使用kubeadm快速搭建k8s集群学习

# 系统初始化配置 **部署Kubernetes(k8s)时，为什么要关闭swap、selinux、防火墙？** - Swap会导致docker的运行不正常，性能下降，是个bug，但是后来关闭swap就解决了，就变成了通用方案，后续可能修复了(我没关注)，基本上默认关闭了就OK，内存开大点儿不太会oom，本来容器也可以限制内存的使用量，控制一下就好。 - Selinux是内核级别的一个安全模块，通过安全上下文的方式控制应用服务的权限，是应用和操作系统之间的一道ACL，但不是所有的程序都会去适配这个模块，不适配的话开着也不起作用，何况还有规则影响到正常的服务。比如权限等等相关的问题。 - 防火墙看你说的是那种，如果你说的是iptables的话 ，那样应该是开着的，不应该关。k8s通过iptables做pod间流量的转发和端口映射，如果你说的防火墙是firewalld那就应该关闭，因为firewalld和iptables属于前后两代方案，互斥。当然，如果k8s没用iptables，比如用了其他的方式那就不需要用到自然可以关掉。 ## 配置主机名和hosts 配置所有节点的主机名 ```bash # 192.168.99.100 Master节点 [root@localhost ~]# hostnamectl set-hostname k8s-master [root@localhost ~]# su [root@k8s-master ~]# # 192.168.99.201 Node1节点 [root@localhost ~]# hostnamectl set-hostname k8s-node1 [root@localhost ~]# su [root@k8s-node1 ~]# # 192.168.99.202 Node2节点 [root@localhost ~]# hostnamectl set-hostname k8s-node2 [root@localhost ~]# su [root@k8s-node2 ~]# ``` 配置hosts，以Master节点为例，其他一样 ```bash [root@k8s-master ~]# vim /etc/hosts [root@k8s-master ~]# cat /etc/hosts | grep k8s 192.168.99.100 k8s-master 192.168.99.201 k8s-node1 192.168.99.202 k8s-node2 # 之后ping主机名就可以通了 [root@k8s-master ~]# ping k8s-node2 ``` ## 关闭防火墙 所有节点都需要关闭 ```bash # 关闭防火墙 [root@k8s-master ~]# systemctl stop firewalld # 禁止开机启动 [root@k8s-master ~]# systemctl disable firewalld Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. ``` ## 关闭SELinux 关闭selinux：linux下的一个安全机制，主要是对文件系统访问做一个权限控制，这个权限控制会影响到kubernetes中的一个组件kuberlete的安装，因为这个组件的安装会访问本地的文件操作系统。 所有节点都需要操作 ```bash # 先临时关闭，永久关闭需要重启主机 [root@k8s-master ~]# setenforce 0 # 永久关闭 [root@k8s-master ~]# sed -i 's/enforcing/disabled/' /etc/selinux/config [root@k8s-master ~]# cat /etc/selinux/config | grep ^SELINUX= SELINUX=disabled ``` ## 关闭Swap 禁止swap分区：swap分区的作用是当物理内存不足时，利用swap分区做数据交换，但是在kubernetes中完全不支持swap分区，所以必须禁止掉，或者创建系统的时候就不创建它。 所有节点都需要关闭 ```bash # 临时关闭 [root@k8s-master ~]# swapoff -a # 永久关闭 [root@k8s-master ~]# cat /etc/fstab | grep swap /dev/mapper/centos-swap swap swap defaults 0 0 [root@k8s-master ~]# vim /etc/fstab [root@k8s-master ~]# cat /etc/fstab | grep swap # /dev/mapper/centos-swap swap swap defaults 0 0 ``` ## 将桥接的IPv4流量传递到iptables的链 有一些ipv4的流量不能走iptables链，（linux内核的一个过滤器，每个流量都会经过他，然后再匹配是否可进入当前应用进程去处理），导致流量丢失。 ```bash # k8s.conf文件原来不存在，需要自己创建的 [root@k8s-master ~]# cat > /etc/sysctl.d/k8s.conf << EOF > net.bridge.bridge-nf-call-ip6tables = 1 > net.bridge.bridge-nf-call-iptables = 1 > EOF [root@k8s-master ~]# sysctl --system * Applying /usr/lib/sysctl.d/00-system.conf ... * Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ... kernel.yama.ptrace_scope = 0 * Applying /usr/lib/sysctl.d/50-default.conf ... kernel.sysrq = 16 kernel.core_uses_pid = 1 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.default.promote_secondaries = 1 net.ipv4.conf.all.promote_secondaries = 1 fs.protected_hardlinks = 1 fs.protected_symlinks = 1 * Applying /etc/sysctl.d/99-sysctl.conf ... * Applying /etc/sysctl.d/k8s.conf ... * Applying /etc/sysctl.conf ... ``` # 所有节点安装Docker、kubeadm、kubelet、kubectl 安装安装Doctor、kubeadm【引导集群的客户端工具】、kubelet【kubernetes中管理容器】 ## 安装Docker（实际版本18.06.3） ```bash # 配置Docker官方源并将下载的源文件存放于/etc/yum.repos.d/下 [root@k8s-master ~]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo # 安装Docker-CE，先测试最新版，如果指定的版本号错误，会安装最新版，先测试下行不行 [root@k8s-master ~]# yum install docker-ce-19.03.9-3.el7 -y # 实际需要安装 [root@k8s-master ~]# yum install docker-ce-18.06.3.ce-3.el7 # 启动Docker [root@k8s-master ~]# systemctl start docker [root@k8s-master ~]# systemctl enable docker Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service. # 查看Docker版本 [root@k8s-master ~]# docker --version Docker version 19.03.9, build 9d988398e7 ``` ## 安装kubeadm，kubelet和kubectl（实际版本1.18.2） ubernetes默认CRI（容器运行时）为Docker，因此先安装Docker。 ```bash # 设置源 [root@k8s-master ~]# cat > /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF # 开始安装 [root@k8s-master ~]# yum install kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2 -y # 提示xxx.rpm公钥尚未安装 # 跳过公钥检查 [root@k8s-master ~]# yum install kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2 --nogpgcheck -y 已安装: kubeadm.x86_64 0:1.18.2-0 kubectl.x86_64 0:1.18.2-0 kubelet.x86_64 0:1.18.2-0 作为依赖被安装: conntrack-tools.x86_64 0:1.4.4-7.el7 cri-tools.x86_64 0:1.13.0-0 kubernetes-cni.x86_64 0:0.7.5-0 libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7 完毕！ [root@k8s-master ~]# whereis kubeadm kubeadm: /usr/bin/kubeadm [root@k8s-master ~]# whereis kubectl kubectl: /usr/bin/kubectl [root@k8s-master ~]# whereis kubelet kubelet: /usr/bin/kubelet # 此时此刻不需要启动kubelet，因为最后kubeadm会一键部署，仅仅设置开机启动即可 [root@k8s-master ~]# systemctl enable kubelet Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. ``` # 部署Kubernetes Master ## 参数说明 `kubeadm init`命令参数说明 ```bash --apiserver-advertise-address string 设置 apiserver 绑定的 IP. --apiserver-bind-port int32 设置apiserver 监听的端口. (默认 6443) --apiserver-cert-extra-sans strings api证书中指定额外的Subject Alternative Names (SANs) 可以是IP 也可以是DNS名称。 证书是和SAN绑定的。 --cert-dir string 证书存放的目录 (默认 "/etc/kubernetes/pki") --certificate-key string kubeadm-cert secret 中 用于加密 control-plane 证书的key --config string kubeadm 配置文件的路径. --cri-socket string CRI socket 文件路径，如果为空 kubeadm 将自动发现相关的socket文件; 只有当机器中存在多个 CRI socket 或者 存在非标准 CRI socket 时才指定. --dry-run 测试，并不真正执行;输出运行后的结果. --feature-gates string 指定启用哪些额外的feature 使用 key=value 对的形式。 -h, --help 帮助文档 --ignore-preflight-errors strings 忽略前置检查错误，被忽略的错误将被显示为警告. 例子: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. --image-repository string 选择拉取 control plane images 的镜像repo (default "k8s.gcr.io") --kubernetes-version string 选择K8S版本. (default "stable-1") --node-name string 指定node的名称，默认使用 node 的 hostname. --pod-network-cidr string 指定 pod 的网络， control plane 会自动将 网络发布到其他节点的node，让其上启动的容器使用此网络 --service-cidr string 指定service 的IP 范围. (default "10.96.0.0/12") --service-dns-domain string 指定 service 的 dns 后缀, e.g. "myorg.internal". (default "cluster.local") --skip-certificate-key-print 不打印 control-plane 用于加密证书的key. --skip-phases strings 跳过指定的阶段（phase） --skip-token-print 不打印 kubeadm init 生成的 default bootstrap token --token string 指定 node 和control plane 之间，简历双向认证的token ，格式为 [a-z0-9]{6}\.[a-z0-9]{16} - e.g. abcdef.0123456789abcdef --token-ttl duration token 自动删除的时间间隔。 (e.g. 1s, 2m, 3h). 如果设置为 '0', token 永不过期 (default 24h0m0s) --upload-certs 上传 control-plane 证书到 kubeadm-certs Secret. ``` ## 在Master节点kubeadm部署 ### kubeadm init初始化命令 由于测试修改了`kubernetes-version`，但是阿里云没有`v1.18.3 not found`，仍使用`v1.18.2` ```bash [root@k8s-master ~]# kubeadm init \ --apiserver-advertise-address=192.168.99.100 \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.18.2 \ --service-cidr=10.1.0.0/16 \ --pod-network-cidr=10.244.0.0/16 ``` 由于默认拉取镜像地址`k8s.gcr.io`国内无法访问，这里指定阿里云镜像仓库地址。 ### 初始化报错处理 ```bash W0523 16:43:11.953762 3851 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] [init] Using Kubernetes version: v1.18.2 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2 [ERROR CRI]: container runtime is not running: output: Client: Debug Mode: false Server: ERROR: Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.39 errors pretty printing info , error: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher ``` #### client version 1.40 is too new Docker版本 19.03.9 过高，考虑升级k8s版本，从 kubernetes 1.18.2 -> 1.18.3，仍然不行，在降级Docker ```bash [root@k8s-master ~]# yum update kubelet-1.18.3 kubeadm-1.18.3 kubectl-1.18.3 --nogpgcheck -y 更新完毕: kubeadm.x86_64 0:1.18.3-0 kubectl.x86_64 0:1.18.3-0 kubelet.x86_64 0:1.18.3-0 完毕！ ``` #### docker 19.03.9降级18.06.3安装冲突 ```bash # 降级Docker版本 [root@k8s-master ~]# yum list docker-ce --showduplicates | sort -r 已加载插件：fastestmirror 已安装的软件包 可安装的软件包 * updates: mirror.lzu.edu.cn Loading mirror speeds from cached hostfile * extras: mirror.lzu.edu.cn docker-ce.x86_64 3:19.03.9-3.el7 docker-ce-stable docker-ce.x86_64 3:19.03.8-3.el7 docker-ce-stable docker-ce.x86_64 3:19.03.7-3.el7 docker-ce-stable docker-ce.x86_64 3:19.03.6-3.el7 docker-ce-stable docker-ce.x86_64 3:19.03.5-3.el7 docker-ce-stable docker-ce.x86_64 3:19.03.4-3.el7 docker-ce-stable docker-ce.x86_64 3:19.03.3-3.el7 docker-ce-stable docker-ce.x86_64 3:19.03.2-3.el7 docker-ce-stable docker-ce.x86_64 3:19.03.1-3.el7 docker-ce-stable docker-ce.x86_64 3:19.03.0-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.9-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.9-3.el7 @docker-ce-stable docker-ce.x86_64 3:18.09.8-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.7-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.6-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.5-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.4-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.3-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.2-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.1-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.0-3.el7 docker-ce-stable docker-ce.x86_64 18.06.3.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.2.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.1.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.0.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.03.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 18.03.0.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.12.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.12.0.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.09.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.09.0.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.06.2.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.06.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.06.0.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.03.3.ce-1.el7 docker-ce-stable docker-ce.x86_64 17.03.2.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.03.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.03.0.ce-1.el7.centos docker-ce-stable * base: mirror.lzu.edu.cn [root@k8s-master ~]# yum remove docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-selinux \ docker-engine-selinux \ docker-engine [root@k8s-master ~]# yum install docker-ce-18.06.3.ce-3.el7 # 安装报错 Transaction check error: file /usr/bin/docker from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.9-3.el7.x86_64 file /usr/share/bash-completion/completions/docker from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.9-3.el7.x86_64 file /usr/share/fish/vendor_completions.d/docker.fish from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.9-3.el7.x86_64 file /usr/share/man/man1/docker-attach.1.gz from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.9-3.el7.x86_64 file /usr/share/man/man1/docker-checkpoint-create.1.gz from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.9-3.el7.x86_64 file /usr/share/man/man1/docker-checkpoint-ls.1.gz from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.9-3.el7.x86_64 # 。。。。 # 可以通过find / -name docker查找缓存包，如果找不到直接下载即可。 # 用rpm手动安装并添加–replacefiles 参数 [root@k8s-master ~]# rpm -ivh /var/cache/yum/x86_64/7/docker-ce-stable/packages/docker-ce-18.06.3.ce-3.el7.x86_64.rpm --replacefiles 错误：依赖检测失败： container-selinux >= 2.9 被 docker-ce-18.06.3.ce-3.el7.x86_64 需要 # 需要安装container-selinux [root@k8s-master ~]# yum install container-selinux -y 已安装: container-selinux.noarch 2:2.119.1-1.c57a6f9.el7 完毕！ # 再次使用rpm安装docker [root@k8s-master ~]# rpm -ivh /var/cache/yum/x86_64/7/docker-ce-stable/packages/docker-ce-18.06.3.ce-3.el7.x86_64.rpm --replacefiles [root@k8s-master ~]# docker version Client: Version: 18.06.3-ce API version: 1.38 # 启动docker [root@k8s-master ~]# systemctl start docker [root@k8s-master ~]# systemctl enable docker Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service. ``` #### cgroupfs警告问题（所有节点修改） [警告IsDockerSystemdCheck]：检测到“cgroupfs”作为Docker cgroup驱动程序。 推荐的驱动程序是“systemd”。 ```bash [root@k8s-node1 ~]# vim /etc/docker/daemon.json [root@k8s-node1 ~]# cat /etc/docker/daemon.json { "exec-opts":["native.cgroupdriver=systemd"] } [root@k8s-node1 ~]# systemctl restart docker ``` #### CPU核心数修改 由于是VMware安装的，所以所有节点修改 处理器内核数量大于2 #### 阿里云v1.18.3 not found ```bash [root@k8s-master ~]# kubeadm init \ --apiserver-advertise-address=192.168.99.100 \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.18.3 \ --service-cidr=10.1.0.0/16\ --pod-network-cidr=10.244.0.0/16 rror execution phase preflight: [preflight] Some fatal errors occurred: [ERROR ImagePull]: failed to pull image registry.aliyuncs.com/google_containers/kube-apiserver:v1.18.3: output: Error response from daemon: manifest for registry.aliyuncs.com/google_containers/kube-apiserver:v1.18.3 not found , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.aliyuncs.com/google_containers/kube-controller-manager:v1.18.3: output: Error response from daemon: manifest for registry.aliyuncs.com/google_containers/kube-controller-manager:v1.18.3 not found , error: exit status 1 # 所有节点上都进行降级安装，即 1.18.3 -> 1.18.2 [root@k8s-master ~]# yum downgrade kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2 --nogpgcheck -y ``` ### kubeadm init初始化日志 ```bash # 1、运行一系列预检代码来检查系统的状态； # 大部分的检查只会抛出一个警告，也有一部分会抛出异常错误从而导致工作流推出 # （比如没有关闭swap或者没有安装docker）。官方给出一个参数–ignore-preflight-errors=... W0523 18:08:32.999414 2448 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] [init] Using Kubernetes version: v1.18.2 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet # 2、生成一个用来认证k8s组件间调用的自签名的CA（Certificate Authority，证书授权）； # 这个证书也可以通过–cert-dir（默认是/etc/kubernetets/pki）的方式传入，那么这一步就会跳过。 [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.1.0.1 192.168.99.100] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.99.100 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.99.100 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key # 3、把kubelet、controller-manager和scheduler等组件的配置文件写到/etc/kubernets/目录， # 这几个组件会使用这些配置文件来连接API-server的服务； # 除了上面几个配置文件，还会生成一个管理相关的admin.conf文件。 [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file # 4、创建一些 静态pod 的配置文件了，包括API-server、controller-manager和scheduler。 # 假如没有提供外部etcd，还会另外生成一个etcd的静态Pod配置文件。 # 这些静态pod会被写入/etc/kubernetes/manifests，kubelet进程会监控这个目录，从而创建相关的pod。 [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" W0523 18:09:33.958611 2448 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [control-plane] Creating static Pod manifest for "kube-scheduler" W0523 18:09:33.960089 2448 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" # 5、如果上一步比较顺利，这个时候k8s的控制面进程（api-server、controller-manager、scheduler）就全都起来了。 [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 16.002340 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs # 6、给当前的节点（Master节点）打label和taints，从而防止其他的负载在这个节点运行。 [mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] # 7、生成token，其他节点如果想加入当前节点（Master）所在的k8s集群，会用到这个token。 [bootstrap-token] Using token: mfouno.1nacdahemjebmzgb # 8、进行一些允许节点以 Bootstrap Tokens) 和 TLS bootstrapping 方式加入集群的必要的操作 # 设置RBAC规则，同时创建一个用于节点加入集群的ConfigMap（包含了加入集群需要的所有信息）。 # 让Bootstrap Tokens可以访问CSR签名的API。 # 给新的CSR请求配置自动认证机制。 [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key # 9、通过API-server安装DNS服务器（1.11版本后默认为CoreDNS，早期版本默认为kube-dns）和kube-proxy插件。 # 这里需要注意的是，DNS服务器只有在安装了CNI（flannel或calico）之后才会真正部署，否则会处于挂起（pending）状态。 [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.99.100:6443 --token mfouno.1nacdahemjebmzgb \ --discovery-token-ca-cert-hash sha256:b333ae5c7fc2888d4a416816b1c977866277caf467e1a0f18e585c857d5d17ed ``` ### 开始使用集群前创建配置文件 直接复制上面的一段即可 ```bash [root@k8s-master ~]# mkdir -p $HOME/.kube [root@k8s-master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@k8s-master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` 使用`kubectl`获取节点 ```bash [root@k8s-master ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master NotReady master 67m v1.18.2 ``` 上面的状态为`NotReady`，因为他在等待网络的加入。 ### 安装Pod网络插件（CNI） #### flannel解释 [Kubernetes-基于flannel的集群网络](https://www.kubernetes.org.cn/4105.html) [Kubernetes网络分析之Flannel](https://www.kubernetes.org.cn/4887.html) [flannel is a network fabric for containers, designed for Kubernetes（GitHub）](https://github.com/coreos/flannel) #### 应用kube-flannel.yml文件 Master节点，Node节点加入后自动下载 ```bash # 应用在线的kube-flannel.yml文件 [root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds-amd64 created daemonset.apps/kube-flannel-ds-arm64 created daemonset.apps/kube-flannel-ds-arm created daemonset.apps/kube-flannel-ds-ppc64le created daemonset.apps/kube-flannel-ds-s390x created # 查看命名空间kube-system中的pod变化 [root@k8s-master ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-7ff77c879f-5dl9p 0/1 Pending 0 74m coredns-7ff77c879f-x5cmh 0/1 Pending 0 74m etcd-k8s-master 1/1 Running 0 74m kube-apiserver-k8s-master 1/1 Running 0 74m kube-controller-manager-k8s-master 1/1 Running 0 74m kube-flannel-ds-amd64-d7mbx 0/1 Init:0/1 0 55s kube-proxy-jndns 1/1 Running 0 74m kube-scheduler-k8s-master 1/1 Running 0 74m # 节点只有master，因为还没有其他节点加入进来 [root@k8s-master ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master NotReady master 75m v1.18.2 [root@k8s-master ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-7ff77c879f-5dl9p 0/1 Pending 0 80m coredns-7ff77c879f-x5cmh 0/1 Pending 0 80m etcd-k8s-master 1/1 Running 0 80m kube-apiserver-k8s-master 1/1 Running 0 80m kube-controller-manager-k8s-master 1/1 Running 0 80m kube-flannel-ds-amd64-d7mbx 0/1 Init:ImagePullBackOff 0 6m18s kube-proxy-jndns 1/1 Running 0 80m kube-scheduler-k8s-master 1/1 Running 0 80m # 查看所有pod的状态 [root@k8s-master ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-7ff77c879f-5dl9p 1/1 Running 0 3h5m coredns-7ff77c879f-x5cmh 1/1 Running 0 3h5m etcd-k8s-master 1/1 Running 0 3h5m kube-apiserver-k8s-master 1/1 Running 0 3h5m kube-controller-manager-k8s-master 1/1 Running 0 3h5m kube-flannel-ds-amd64-d7mbx 1/1 Running 0 111m kube-proxy-jndns 1/1 Running 0 3h5m kube-scheduler-k8s-master 1/1 Running 0 3h5m ``` #### kube-flannel下载慢处理 如果下载太慢，将 https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 下载到本地 修改，将image的地址修改为Docker Hub国内仓库地址，或者私有仓库。 ```yaml containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.12.0-amd64 ``` 例如修改为七牛云的，然后再次执行`kubectl apply -f kube-flannel.yml`，`kube-flannel.yml`为本地的文件。 ```yaml containers: - name: kube-flannel image: quay-mirror.qiniu.com/coreos/flannel:v0.12.0-amd64 ``` 编辑该文件，然后重新应用`kubectl apply -f kube-flannel.yml`，即创建+更新，可以重复使用 ```bash [root@k8s-master ~]# vim kube-flannel.yml # 重新应用更新的本地kube-flannel.yml文件 [root@k8s-master ~]# kubectl apply -f kube-flannel.yml podsecuritypolicy.policy/psp.flannel.unprivileged configured clusterrole.rbac.authorization.k8s.io/flannel unchanged clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged serviceaccount/flannel unchanged configmap/kube-flannel-cfg unchanged daemonset.apps/kube-flannel-ds-amd64 configured # 可以看到这项更新了 daemonset.apps/kube-flannel-ds-arm64 unchanged daemonset.apps/kube-flannel-ds-arm unchanged daemonset.apps/kube-flannel-ds-ppc64le unchanged daemonset.apps/kube-flannel-ds-s390x unchanged ``` # 将Node加入集群 在所有Node节点上运行，向集群添加新节点，执行在`kubeadm init`输出的`kubeadm join`命令 为Node节点初始化 ## Node1节点加入集群 ```bash [root@k8s-node1 ~]# kubeadm join 192.168.99.100:6443 --token mfouno.1nacdahemjebmzgb \ --discovery-token-ca-cert-hash sha256:b333ae5c7fc2888d4a416816b1c977866277caf467e1a0f18e585c857d5d17ed # Node1日志如下： W0523 21:21:28.246095 10953 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ``` ## Node2节点加入集群 ```bash [root@k8s-node2 ~]# kubeadm join 192.168.99.100:6443 --token mfouno.1nacdahemjebmzgb \ --discovery-token-ca-cert-hash sha256:b333ae5c7fc2888d4a416816b1c977866277caf467e1a0f18e585c857d5d17ed # Node2日志如下 W0523 21:23:01.585236 11009 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ``` ## Master查看集群状态 ### 查看集群中的节点信息 ```bash [root@k8s-master ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master Ready master 3h13m v1.18.2 k8s-node1 NotReady <none> 119s v1.18.3 k8s-node2 NotReady <none> 26s v1.18.3 # 过一段时间后，有一个节点已经加入进来了 [root@k8s-master ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master Ready master 3h32m v1.18.2 k8s-node1 NotReady <none> 21m v1.18.3 k8s-node2 Ready <none> 19m v1.18.3 ``` ### 查看命名空间kube-system下的所有pod信息 ```bash [root@k8s-master ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-7ff77c879f-5dl9p 1/1 Running 0 3h21m coredns-7ff77c879f-x5cmh 1/1 Running 0 3h21m etcd-k8s-master 1/1 Running 0 3h21m kube-apiserver-k8s-master 1/1 Running 0 3h21m kube-controller-manager-k8s-master 1/1 Running 0 3h21m kube-flannel-ds-amd64-d7mbx 1/1 Running 0 127m kube-flannel-ds-amd64-lbqwz 0/1 Init:ImagePullBackOff 0 9m39s kube-flannel-ds-amd64-z69q4 0/1 Init:ImagePullBackOff 0 8m6s kube-proxy-dwpvg 1/1 Running 0 9m39s kube-proxy-jndns 1/1 Running 0 3h21m kube-proxy-w88h4 1/1 Running 0 8m6s kube-scheduler-k8s-master 1/1 Running 0 3h21m # # 过一段时间后，有一个节点已经加入进来了 [root@k8s-master ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-7ff77c879f-5dl9p 1/1 Running 0 3h32m coredns-7ff77c879f-x5cmh 1/1 Running 0 3h32m etcd-k8s-master 1/1 Running 0 3h32m kube-apiserver-k8s-master 1/1 Running 0 3h32m kube-controller-manager-k8s-master 1/1 Running 0 3h32m kube-flannel-ds-amd64-d7mbx 1/1 Running 0 138m kube-flannel-ds-amd64-th9h9 1/1 Running 0 3m44s kube-flannel-ds-amd64-xv27n 0/1 Init:ImagePullBackOff 0 3m44s kube-proxy-dwpvg 1/1 Running 0 21m kube-proxy-jndns 1/1 Running 0 3h32m kube-proxy-w88h4 1/1 Running 0 19m kube-scheduler-k8s-master 1/1 Running 0 3h32m ``` # Master节点测试kubernetes集群 ## 创建一个nginx的资源 ```bash # 创建一个nginx的pod [root@k8s-master ~]# kubectl create deployment nginx --image=nginx deployment.apps/nginx created # 创建service：暴露容器让外部访问，--type=NodePort说明创建的是NodePort类型，则通过任意一个NodeIP+Port就可以访问。 [root@k8s-master ~]# kubectl expose deployment nginx --port=80 --type=NodePort service/nginx exposed ``` ## 查看pod、svc信息 `-o wide` 选项可以查看存在哪个对应的节点 ```bash # 查看pod详情 [root@k8s-master ~]# kubectl get pod,svc NAME READY STATUS RESTARTS AGE pod/nginx-f89759699-vn4gb 0/1 ImagePullBackOff 0 32s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 3h35m service/nginx NodePort 10.1.0.141 <none> 80:31519/TCP 25s [root@k8s-master ~]# kubectl get pod,svc -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/nginx-f89759699-vn4gb 0/1 ImagePullBackOff 0 49s 10.244.3.2 k8s-node2 <none> <none> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 3h35m <none> service/nginx NodePort 10.1.0.141 <none> 80:31519/TCP 42s app=nginx ``` ## 查看pod的详情描述 状态为`ImagePullBackOff`或`ErrImagePull`时可以查看该pod描述，在排错时尤为实用 ```bash [root@k8s-master ~]# kubectl describe pod nginx-f89759699-vn4gb Name: nginx-f89759699-vn4gb Namespace: default Priority: 0 Node: k8s-node2/192.168.99.202 Start Time: Sat, 23 May 2020 21:44:55 +0800 Labels: app=nginx pod-template-hash=f89759699 Annotations: <none> Status: Pending IP: 10.244.3.2 IPs: IP: 10.244.3.2 Controlled By: ReplicaSet/nginx-f89759699 Containers: nginx: Container ID: Image: nginx Image ID: Port: <none> Host Port: <none> State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-s2qmv (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: default-token-s2qmv: Type: Secret (a volume populated by a Secret) SecretName: default-token-s2qmv Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 6m31s default-scheduler Successfully assigned default/nginx-f89759699-vn4gb to k8s-node2 Warning Failed 5m47s kubelet, k8s-node2 Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/library/nginx/manifests/latest: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fnginx%3Apull&service=registry.docker.io: net/http: TLS handshake timeout Normal Pulling 4m21s (x4 over 6m30s) kubelet, k8s-node2 Pulling image "nginx" Warning Failed 4m8s (x3 over 6m20s) kubelet, k8s-node2 Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: TLS handshake timeout Warning Failed 4m8s (x4 over 6m20s) kubelet, k8s-node2 Error: ErrImagePull Warning Failed 3m41s (x7 over 6m19s) kubelet, k8s-node2 Error: ImagePullBackOff Normal BackOff 78s (x14 over 6m19s) kubelet, k8s-node2 Back-off pulling image "nginx" ``` ## 拉取Docker镜像慢处理 所有节点都设置国内的源 ```bash [root@k8s-master ~]# vim /etc/docker/daemon.json [root@k8s-master ~]# cat /etc/docker/daemon.json { "exec-opts":["native.cgroupdriver=systemd"], "registry-mirrors": [ "https://hub-mirror.c.163.com", "https://docker.mirrors.ustc.edu.cn" ] } [root@k8s-master ~]# systemctl restart docker [root@k8s-master ~]# docker info # ... Registry Mirrors: https://hub-mirror.c.163.com/ https://docker.mirrors.ustc.edu.cn/ # ... ``` 稍等下再次进行查看 ### 查看pod、svc信息 ```bash [root@k8s-master ~]# kubectl get pod,svc NAME READY STATUS RESTARTS AGE pod/nginx-f89759699-vn4gb 1/1 Running 0 2d22h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 3d2h service/nginx NodePort 10.1.0.141 <none> 80:31519/TCP 2d22h ``` 可以看到资源`nginx-f89759699-vn4gb`的状态由`ImagePullBackOff`变为`Running`。 #### 访问nginx资源 现在就可以通过任意一个节点+`31519`端口访问到，例如`http://192.168.99.100:31519/`、`http://192.168.99.201:31519/`、`http://192.168.99.202:31519/`  ### 查看该pod的详情 ```bash [root@k8s-master ~]# kubectl describe pod nginx-f89759699-vn4gb Name: nginx-f89759699-vn4gb Namespace: default Priority: 0 Node: k8s-node2/192.168.99.202 Start Time: Sat, 23 May 2020 21:44:55 +0800 Labels: app=nginx pod-template-hash=f89759699 Annotations: <none> Status: Running IP: 10.244.3.5 # ..... Warning FailedCreatePodSandBox 107s kubelet, k8s-node2 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1953fda080b8d84918c671364c50409e3365a34dd1f983e9b2dc4f3042b96e71" network for pod "nginx-f89759699-vn4gb": networkPlugin cni failed to set up pod "nginx-f89759699-vn4gb_default" network: open /run/flannel/subnet.env: no such file or directory Normal Pulling 106s kubelet, k8s-node2 Pulling image "nginx" Normal Pulled 82s kubelet, k8s-node2 Successfully pulled image "nginx" Normal Created 82s kubelet, k8s-node2 Created container nginx Normal Started 82s kubelet, k8s-node2 Started container nginx ``` ### 查看集群节点是否正常启动了 ```bash [root@k8s-master ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master Ready master 3d2h v1.18.2 k8s-node1 Ready <none> 2d23h v1.18.3 k8s-node2 Ready <none> 2d23h v1.18.3 [root@k8s-master ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-7ff77c879f-5dl9p 1/1 Running 3 3d2h coredns-7ff77c879f-x5cmh 1/1 Running 3 3d2h etcd-k8s-master 1/1 Running 3 3d2h kube-apiserver-k8s-master 1/1 Running 4 3d2h kube-controller-manager-k8s-master 1/1 Running 3 3d2h kube-flannel-ds-amd64-dsf69 1/1 Running 0 30m kube-flannel-ds-amd64-th9h9 1/1 Running 5 2d23h kube-flannel-ds-amd64-xv27n 1/1 Running 0 2d23h kube-proxy-dwpvg 1/1 Running 3 2d23h kube-proxy-jndns 1/1 Running 3 3d2h kube-proxy-w88h4 1/1 Running 4 2d23h kube-scheduler-k8s-master 1/1 Running 4 3d2h ``` ## 扩容 一个可以有多个副本，可以对该资源进行扩容 ```bash [root@k8s-master ~]# kubectl scale deployment nginx --replicas=3 deployment.apps/nginx scaled ``` 通过`--replicas=3`可以指定扩容后有多少个 查看扩容后的pod变化 ```bash # 扩容前 [root@k8s-master ~]# kubectl get pod NAME READY STATUS RESTARTS AGE nginx-f89759699-vn4gb 1/1 Running 0 2d22h # 扩容后 [root@k8s-master ~]# kubectl get pod NAME READY STATUS RESTARTS AGE nginx-f89759699-lm824 1/1 Running 0 14s nginx-f89759699-nf9f4 0/1 ContainerCreating 0 14s nginx-f89759699-vn4gb 1/1 Running 0 2d23h [root@k8s-master ~]# kubectl get pod NAME READY STATUS RESTARTS AGE nginx-f89759699-lm824 1/1 Running 0 108s nginx-f89759699-nf9f4 1/1 Running 0 108s nginx-f89759699-vn4gb 1/1 Running 0 2d23h # 可以看到每个nginx所在的node节点 [root@k8s-master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-f89759699-lm824 1/1 Running 0 3m17s 10.244.3.6 k8s-node2 <none> <none> nginx-f89759699-nf9f4 1/1 Running 0 3m17s 10.244.1.2 k8s-node1 <none> <none> nginx-f89759699-vn4gb 1/1 Running 0 2d23h 10.244.3.5 k8s-node2 <none> <none> ``` 并发性扩大了3倍，对于用户来说，只需要访问之前的端口即可。 # Master部署Dashboard ## 部署dashboard/v1.10.1 ### 下载修改kubernetes-dashboard.yaml文件 ```bash [root@k8s-master ~]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml ``` 如果镜像访问慢，在Docker Hub找到了[kubernetes-dashboard的链接](https://hub.docker.com/r/siriuszg/kubernetes-dashboard-amd64)，将 ```yaml containers: - name: kubernetes-dashboard image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1 ``` 修改为 ```yaml containers: - name: kubernetes-dashboard image: siriuszg/kubernetes-dashboard-amd64:v1.10.1 ``` ### 允许外部访问端口 默认Dashboard只能集群内部访问，修改`Service`为`NodePort`类型，暴露到外部： 将 ```yaml # ------------------- Dashboard Service ------------------- # kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kube-system spec: ports: - port: 443 targetPort: 8443 selector: k8s-app: kubernetes-dashboard ``` 修改为`type:NodePort`、`nodePort: 30001`（端口默认30000起），修改后 ```yaml # ------------------- Dashboard Service ------------------- # kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kube-system spec: type: NodePort ports: - port: 443 targetPort: 8443 nodePort: 30001 selector: k8s-app: kubernetes-dashboard ``` ### 应用kubernetes-dashboard.yaml文件 ```bash [root@k8s-master ~]# kubectl apply -f kubernetes-dashboard.yaml secret/kubernetes-dashboard-certs created serviceaccount/kubernetes-dashboard created role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created deployment.apps/kubernetes-dashboard created error: error parsing kubernetes-dashboard.yaml: error converting YAML to JSON: yaml: line 12: mapping values are not allowed in this context # 每个属性如设置值的时候，属性 冒号 : 后面 要加上空格隔开，然后再写值。检查刚才修改的内容 # 修改后再次执行 [root@k8s-master ~]# kubectl apply -f kubernetes-dashboard.yaml secret/kubernetes-dashboard-certs unchanged serviceaccount/kubernetes-dashboard unchanged role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal unchanged rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal unchanged deployment.apps/kubernetes-dashboard unchanged service/kubernetes-dashboard created ``` ### 查看命名空间kube-system下的pod 默认放在了`kube-system`命名空间下。 如果想要快速运行，可以先在每个节点下使用`docker pull siriuszg/kubernetes-dashboard-amd64:v1.10.1`提前下载好镜像。然后使用`kubectl apply`时不会再次拉取镜像。 ```bash [root@k8s-master ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-7ff77c879f-5dl9p 1/1 Running 3 3d4h coredns-7ff77c879f-x5cmh 1/1 Running 3 3d4h etcd-k8s-master 1/1 Running 3 3d4h kube-apiserver-k8s-master 1/1 Running 4 3d4h kube-controller-manager-k8s-master 1/1 Running 3 3d4h kube-flannel-ds-amd64-dsf69 1/1 Running 0 112m kube-flannel-ds-amd64-th9h9 1/1 Running 5 3d kube-flannel-ds-amd64-xv27n 1/1 Running 0 3d kube-proxy-dwpvg 1/1 Running 3 3d kube-proxy-jndns 1/1 Running 3 3d4h kube-proxy-w88h4 1/1 Running 4 3d kube-scheduler-k8s-master 1/1 Running 4 3d4h kubernetes-dashboard-694997dccd-htfqx 0/1 ContainerCreating 0 10m # 过了一会儿，就运行起来了 [root@k8s-master ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-7ff77c879f-5dl9p 1/1 Running 3 3d4h coredns-7ff77c879f-x5cmh 1/1 Running 3 3d4h etcd-k8s-master 1/1 Running 3 3d4h kube-apiserver-k8s-master 1/1 Running 4 3d4h kube-controller-manager-k8s-master 1/1 Running 3 3d4h kube-flannel-ds-amd64-dsf69 1/1 Running 0 114m kube-flannel-ds-amd64-th9h9 1/1 Running 5 3d kube-flannel-ds-amd64-xv27n 1/1 Running 0 3d kube-proxy-dwpvg 1/1 Running 3 3d kube-proxy-jndns 1/1 Running 3 3d4h kube-proxy-w88h4 1/1 Running 4 3d kube-scheduler-k8s-master 1/1 Running 4 3d4h kubernetes-dashboard-694997dccd-htfqx 1/1 Running 0 12m # 查看运行的节点信息 [root@k8s-master ~]# kubectl get pod -n kube-system -o wide | grep kubernetes-dashboard kubernetes-dashboard-694997dccd-htfqx 1/1 Running 0 15m 10.244.1.3 k8s-node1 <none> <none> ``` ### 查看暴露的端口 ```bash [root@k8s-master ~]# kubectl get pod,svc -n kube-system NAME READY STATUS RESTARTS AGE pod/coredns-7ff77c879f-5dl9p 1/1 Running 3 3d4h pod/coredns-7ff77c879f-x5cmh 1/1 Running 3 3d4h pod/etcd-k8s-master 1/1 Running 3 3d4h pod/kube-apiserver-k8s-master 1/1 Running 4 3d4h pod/kube-controller-manager-k8s-master 1/1 Running 3 3d4h pod/kube-flannel-ds-amd64-dsf69 1/1 Running 0 119m pod/kube-flannel-ds-amd64-th9h9 1/1 Running 5 3d pod/kube-flannel-ds-amd64-xv27n 1/1 Running 0 3d pod/kube-proxy-dwpvg 1/1 Running 3 3d pod/kube-proxy-jndns 1/1 Running 3 3d4h pod/kube-proxy-w88h4 1/1 Running 4 3d pod/kube-scheduler-k8s-master 1/1 Running 4 3d4h pod/kubernetes-dashboard-694997dccd-htfqx 1/1 Running 0 18m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kube-dns ClusterIP 10.1.0.10 <none> 53/UDP,53/TCP,9153/TCP 3d4h service/kubernetes-dashboard NodePort 10.1.116.46 <none> 443:30001/TCP 8m31s ``` 可以看到暴露的端口为`30001`。且需要使用https访问： https://192.168.99.100:30001/  在这个页面没有**继续访问**的按钮，直接在键盘上打`thisisunsafe`就可以自动进去了 https://192.168.99.100:30001/#!/login  ## 创建用户token登录 选择令牌，创建一个管理员的身份去访问UI，创建一个面向应用的`serviceaccount`用户去访问 ```bash [root@k8s-master ~]# kubectl create serviceaccount dashboard-admin -n kube-system serviceaccount/dashboard-admin created [root@k8s-master ~]# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin created [root@k8s-master ~]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}') Name: dashboard-admin-token-n25df Namespace: kube-system Labels: <none> Annotations: kubernetes.io/service-account.name: dashboard-admin kubernetes.io/service-account.uid: 074b8084-cc73-4a30-88e2-a59ee518fee1 Type: kubernetes.io/service-account-token Data ==== token: eyJhbGciOiJSUzI1NiIsImtpZCI6IlVHZDZsTGNLbTM3OEUzYzk3Q3k1dTFJVnFEZFJ6eG90cHZZMjN6RUdLVnMifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tbjI1ZGYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMDc0YjgwODQtY2M3My00YTMwLTg4ZTItYTU5ZWU1MThmZWUxIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.FNoMLtHKhmcMwFdEE2PcvBWRV4nei5T8wPwvJWSH_A_H09OZp-FIrwxotswzZbVE90-h76zXLbAghNp-kzPYWvdXYWlMUraFYnNRrRQMsrzcaHe_Ex5KiDwMmavxa0fd_1x0RQLNsIDL20gUR8LVfelKZ2Cdzy0v_xHAQjui4bW2HX6GtSmfVTcBO8PHrCwgOGNw3uBKjNxZcmhMoOc_m4Zj2lerUjpZFpmyhoGbmCytPgGywxPmWK56pK-SsDYLFFX-kSSpeDyv2GvH5MXK51aONU5P3n-jo4OngUvTjFT-FTGmSZ_oKN_YMTlsNlUYjrVHhDWmeqdYHkqGhSgV5Q ca.crt: 1025 bytes namespace: 11 bytes ``` 然后将token粘贴到输入框   ## 处理“未知服务器错误 (404)”问题 提示 ``` 未知服务器错误 (404) the server could not find the requested resource 3 秒内重定向到上一个页面…… ``` 删除使用`kubernetes-dashboard.yaml`旧版创建的资源 ```bash # 通过yaml文件删除 [root@k8s-master ~]# kubectl delete -f kubernetes-dashboard.yaml secret "kubernetes-dashboard-certs" deleted serviceaccount "kubernetes-dashboard" deleted role.rbac.authorization.k8s.io "kubernetes-dashboard-minimal" deleted rolebinding.rbac.authorization.k8s.io "kubernetes-dashboard-minimal" deleted deployment.apps "kubernetes-dashboard" deleted service "kubernetes-dashboard" deleted ``` ## 部署dashboard/v2.0.1 访问 https://github.com/kubernetes/dashboard/releases 安装新版 ### 下载新版recommended.yaml ```bash [root@k8s-master ~]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.1/aio/deploy/recommended.yaml ``` ### 修改recommended.yaml端口暴露 增加`type: NodePort`和`nodePort: 30001` ```yaml --- kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: type: NodePort ports: - port: 443 targetPort: 8443 nodePort: 30001 selector: k8s-app: kubernetes-dashboard ``` ### 应用recommended.yaml 文件 ```bash [root@k8s-master ~]# kubectl apply -f recommended.yaml namespace/kubernetes-dashboard created serviceaccount/kubernetes-dashboard created service/kubernetes-dashboard created secret/kubernetes-dashboard-certs created secret/kubernetes-dashboard-csrf created secret/kubernetes-dashboard-key-holder created configmap/kubernetes-dashboard-settings created role.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created deployment.apps/kubernetes-dashboard created service/dashboard-metrics-scraper created deployment.apps/dashboard-metrics-scraper created ``` 可以看到dashboard是运行到`kubernetes-dashboard`命名空间下的，所以要指定`-n kubernetes-dashboard`查看信息 ### 查看kubernetes-dashboard无法启动的日志 ```bash # 查看状态 [root@k8s-master ~]# kubectl get pod -n kubernetes-dashboard NAME READY STATUS RESTARTS AGE dashboard-metrics-scraper-6b4884c9d5-z4cqb 1/1 Running 7 15m kubernetes-dashboard-7bfbb48676-pgjcg 0/1 CrashLoopBackOff 7 15m # 一直处于CrashLoopBackOff， # 查看详情 [root@k8s-master ~]# kubectl describe pod kubernetes-dashboard-7bfbb48676-pgjcg -n kubernetes-dashboard Name: kubernetes-dashboard-7bfbb48676-pgjcg Namespace: kubernetes-dashboard Priority: 0 Node: k8s-node1/192.168.99.201 Start Time: Wed, 27 May 2020 20:40:51 +0800 Labels: k8s-app=kubernetes-dashboard pod-template-hash=7bfbb48676 # ........ Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned kubernetes-dashboar Normal Pulling 2m26s (x4 over 3m20s) kubelet, k8s-node1 Pulling image "kubernetesui/dashboard:v2. Normal Pulled 2m24s (x4 over 3m17s) kubelet, k8s-node1 Successfully pulled image "kubernetesui/d Normal Created 2m24s (x4 over 3m17s) kubelet, k8s-node1 Created container kubernetes-dashboard Normal Started 2m24s (x4 over 3m17s) kubelet, k8s-node1 Started container kubernetes-dashboard Warning BackOff 115s (x9 over 3m11s) kubelet, k8s-node1 Back-off restarting failed container # 部署到k8s-node1节点上 # 查看日志 [root@k8s-master ~]# kubectl logs kubernetes-dashboard-7bfbb48676-pgjcg -n kubernetes-dashboard 2020/05/27 12:52:00 Starting overwatch 2020/05/27 12:52:00 Using namespace: kubernetes-dashboard 2020/05/27 12:52:00 Using in-cluster config to connect to apiserver 2020/05/27 12:52:00 Using secret token for csrf signing 2020/05/27 12:52:00 Initializing csrf token from kubernetes-dashboard-csrf secret panic: Get https://10.1.0.1:443/api/v1/namespaces/kubernetes-dashboard/secrets/kubernetes-dashboard-csrf: dial tcp 10.1.0.1:443: connect: no route to host goroutine 1 [running]: github.com/kubernetes/dashboard/src/app/backend/client/csrf.(*csrfTokenManager).init(0xc0004ca880) /home/travis/build/kubernetes/dashboard/src/app/backend/client/csrf/manager.go:41 +0x446 github.com/kubernetes/dashboard/src/app/backend/client/csrf.NewCsrfTokenManager(...) /home/travis/build/kubernetes/dashboard/src/app/backend/client/csrf/manager.go:66 github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).initCSRFKey(0xc000462100) /home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:501 +0xc6 github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).init(0xc000462100) /home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:469 +0x47 github.com/kubernetes/dashboard/src/app/backend/client.NewClientManager(...) /home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:550 main.main() /home/travis/build/kubernetes/dashboard/src/app/backend/dashboard.go:105 +0x20d ``` ### 日志中kubernetes-dashboard-csrf报错处理 参考 https://www.jianshu.com/p/e359d3fe238f 的解决方案 由于 pod 不会分配到到 master 节点, 并且 kubeadm部署的 apiserver 中启用的验证方式为 Node 和 RBAC, 且关闭了 `insecure-port`，猜测可能是这个原因导致连接不上 apiServer , 即使是手动修改也不行 `--apiserver-host` 参数也不行。 重新修改 recommended.yaml 文件 ```bash [root@k8s-master ~]# vim recommended.yaml ``` 增加`nodeName: k8s-master`的配置 ```yaml # ... --- kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: kubernetes-dashboard template: metadata: labels: k8s-app: kubernetes-dashboard spec: nodeName: k8s-master containers: - name: kubernetes-dashboard image: kubernetesui/dashboard:v2.0.1 # ... --- kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboard spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: dashboard-metrics-scraper template: metadata: labels: k8s-app: dashboard-metrics-scraper annotations: seccomp.security.alpha.kubernetes.io/pod: 'runtime/default' spec: nodeName: k8s-master containers: - name: dashboard-metrics-scraper image: kubernetesui/metrics-scraper:v1.0.4 # ... ``` 上述配置将指定 `kubernetes-dashboard` 和 `kubernetes-metrics-scraper` 分配部署的节点，此处选择部署在 master 节点解决文中出现的通信问题。之前的端口暴露不变。 ### 重新应用recommended.yaml文件 ```bash [root@k8s-master ~]# kubectl apply -f recommended.yaml ``` 可以看到pod的名称也发生了变化，因为做了配置文件的修改。 ### 查看kubernetes-dashboard状态运行成功 ```bash [root@k8s-master ~]# kubectl get pod -n kubernetes-dashboard NAME READY STATUS RESTARTS AGE dashboard-metrics-scraper-6fb66f87b-gdfbp 1/1 Running 0 3m47s kubernetes-dashboard-fb89c4d57-sz86q 1/1 Running 0 3m47s ``` 访问 https://192.168.99.100:30001/ 在token中输入之前生成的token字符串，点击登录  即可正常进入页面了 

明人不说暗话 领红包就是支持！！！ 来都来了，再到页面底部看下呗~ ↓ ↓ ↓

本文地址： http://blog.starmeow.cn/detail/f7664060cc52bc6f3d620bcedc94a4b6/

很赞哦！ (0)