【K8s+Docker技术全解】04.运维主机部署Harbor环境

## 99.200部署Harbor环境 官网 https://goharbor.io/ 下载 https://github.com/goharbor/harbor/releases 当前最新的为`v1.10.1`，下载`offline`版本`harbor-offline-installer-v1.10.1.tgz` ### 下载Harbor 下载好上传到运维管理服务器 ```bash [root@k8s99-200 certs]# cd [root@k8s99-200 ~]# ls harbor-offline-installer-v1.10.1.tgz # 如果主要指定解压到的目录添加 -C /path/ 即可 [root@k8s99-200 ~]# tar zxf harbor-offline-installer-v1.10.1.tgz -C /opt/ [root@k8s99-200 opt]# ls certs harbor # 软件包管理，用版本号标识，然后用软链接去管理软件包，便于后期升级 [root@k8s99-200 opt]# mv harbor harbor-v1.10.1 [root@k8s99-200 opt]# ln -s harbor-v1.10.1/ harbor [root@k8s99-200 opt]# ls -l 总用量 0 drwxr-xr-x. 2 root root 71 6月 1 20:26 certs lrwxrwxrwx. 1 root root 15 6月 1 20:47 harbor -> harbor-v1.10.1/ drwxr-xr-x. 2 root root 118 6月 1 20:45 harbor-v1.10.1 ``` ### 配置Harbor ```bash [root@k8s99-200 opt]# cd harbor [root@k8s99-200 harbor]# ls common.sh harbor.v1.10.1.tar.gz harbor.yml install.sh LICENSE prepare [root@k8s99-200 harbor]# vim harbor.yml # 修改域名，改成业务域，如果不修改报错：ERROR:root:Please specify hostname hostname: harbor.study.com # 修改http监听端口，避免和其他冲突 http: port: 180 # 注释掉https，否则install报错：ERROR:root:Error: The protocol is https but attribute ssl_cert is not set # https related config # https: # https port for harbor, default is 443 # port: 443 # The path of cert and key files for nginx # certificate: /your/certificate/path # private_key: /your/private/key/path # 默认登录密码，生产环境需要修改密码 harbor_admin_password: Harbor12345 # 数据卷保存位置修改 data_volume: /data/harbor # 日志文件路径，默认如下 log: level: info local: rotate_count: 50 rotate_size: 200M location: /var/log/harbor # 创建数据卷文件夹 [root@k8s99-200 harbor]# mkdir /data/harbor ``` 测试发现可以直接使用`http://harbor.study.com:180/`或`http://IP:180`访问到Harbor 由于Harbor也是使用Docker运行起来的，依赖`docker-compose`，需要进行安装 ```bash [root@k8s99-200 harbor]# yum install docker-compose -y [root@k8s99-200 harbor]# rpm -qa docker-compose docker-compose-1.18.0-4.el7.noarch ``` ### 安装Harbor 完成`docker-compose`安装后，`harbor.yml`配置也处理好了，就可以使用`install.sh`安装了。 ```bash [root@k8s99-200 harbor]# ./install.sh [Step 0]: checking if docker is installed ... Note: docker version: 19.03.10 [Step 1]: checking docker-compose is installed ... Note: docker-compose version: 1.18.0 [Step 2]: loading Harbor images ... # ...省略一些内容 Loaded image: goharbor/clair-adapter-photon:v1.0.1-v1.10.1 Loaded image: goharbor/harbor-jobservice:v1.10.1 Loaded image: goharbor/redis-photon:v1.10.1 Loaded image: goharbor/notary-server-photon:v0.6.1-v1.10.1 Loaded image: goharbor/clair-photon:v2.1.1-v1.10.1 Loaded image: goharbor/harbor-log:v1.10.1 Loaded image: goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.10.1 Loaded image: goharbor/notary-signer-photon:v0.6.1-v1.10.1 Loaded image: goharbor/chartmuseum-photon:v0.9.0-v1.10.1 Loaded image: goharbor/harbor-registryctl:v1.10.1 Loaded image: goharbor/nginx-photon:v1.10.1 Loaded image: goharbor/harbor-migrator:v1.10.1 Loaded image: goharbor/prepare:v1.10.1 Loaded image: goharbor/harbor-portal:v1.10.1 Loaded image: goharbor/harbor-core:v1.10.1 Loaded image: goharbor/harbor-db:v1.10.1 [Step 3]: preparing environment ... [Step 4]: preparing harbor configs ... # ... Creating redis ... done Creating harbor-core ... done Creating network "harborv1101_harbor" with the default driver Creating nginx ... done Creating registryctl ... Creating harbor-db ... Creating harbor-portal ... Creating redis ... Creating registry ... Creating harbor-core ... Creating harbor-jobservice ... Creating nginx ... ✔ ----Harbor has been installed and started successfully.---- # 表示harbor安装完成 ``` 查看安装后的信息 ```bash # 数据目录 [root@k8s99-200 harbor]# ls /data/harbor/ ca_download database job_logs psc redis registry secret # 日志目录 [root@k8s99-200 harbor]# ls /var/log/harbor/ core.log portal.log proxy.log registryctl.log jobservice.log postgresql.log redis.log registry.log [root@k8s99-200 harbor]# docker-compose ps Name Command State Ports -------------------------------------------------------------------------------------- harbor-core /harbor/harbor_core Up harbor-db /docker-entrypoint.sh Up 5432/tcp harbor-jobservice /harbor/harbor_jobservice ... Up harbor-log /bin/sh -c /usr/local/bin/ ... Up 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Up 8080/tcp nginx nginx -g daemon off; Up 0.0.0.0:180->8080/tcp redis redis-server /etc/redis.conf Up 6379/tcp registry /home/harbor/entrypoint.sh Up 5000/tcp registryctl /home/harbor/start.sh Up # 可以看到启动了很多个容器 [root@k8s99-200 harbor]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c1bf60937b54 goharbor/nginx-photon:v1.10.1 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) 0.0.0.0:180->8080/tcp nginx 9aa8b3b112f3 goharbor/harbor-jobservice:v1.10.1 "/harbor/harbor_jobs…" About a minute ago Up About a minute (healthy) harbor-jobservice c7d5c204cded goharbor/harbor-core:v1.10.1 "/harbor/harbor_core" 2 minutes ago Up About a minute (healthy) harbor-core 40dfa4a19491 goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.10.1 "/home/harbor/entryp…" 2 minutes ago Up About a minute (healthy) 5000/tcp registry db3665cea063 goharbor/redis-photon:v1.10.1 "redis-server /etc/r…" 2 minutes ago Up About a minute (healthy) 6379/tcp redis 78842d728086 goharbor/harbor-portal:v1.10.1 "nginx -g 'daemon of…" 2 minutes ago Up About a minute (healthy) 8080/tcp harbor-portal fbc205235efc goharbor/harbor-db:v1.10.1 "/docker-entrypoint.…" 2 minutes ago Up 2 minutes (healthy) 5432/tcp harbor-db a5d80a74ffa1 goharbor/harbor-registryctl:v1.10.1 "/home/harbor/start.…" 2 minutes ago Up 2 minutes (healthy) registryctl 28b9f478861d goharbor/harbor-log:v1.10.1 "/bin/sh -c /usr/loc…" 2 minutes ago Up 2 minutes (healthy) 127.0.0.1:1514->10514/tcp harbor-log ``` 如果机器重启后Harbor容器没有启动完成，可以进`docker-compose.yml`所在目录通过`docker-compose start`进行启动 ```bash [root@k8s99-200 harbor]# docker-compose ps Name Command State Ports ----------------------------------------------------------------------------------------- harbor-core /harbor/harbor_core Exit 255 harbor-db /docker-entrypoint.sh Exit 255 harbor-jobservice /harbor/harbor_jobservice ... Up harbor-log /bin/sh -c /usr/local/bin/ ... Up 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Exit 255 nginx nginx -g daemon off; Exit 255 redis redis-server /etc/redis.conf Up 6379/tcp registry /home/harbor/entrypoint.sh Up 5000/tcp registryctl /home/harbor/start.sh Exit 255 [root@k8s99-200 harbor]# docker-compose start Starting log ... done Starting registry ... done Starting registryctl ... done Starting postgresql ... done Starting portal ... done Starting redis ... done Starting core ... done Starting jobservice ... done Starting proxy ... done ``` 浏览器中可以通过 http://192.168.99.200:180/ 访问到Harbor ### 安装Nginx反代Harbor 要使用Harbor，不能直接使用180端口，需要使用Nginx代理 ```bash [root@k8s99-200 harbor]# yum install nginx -y ``` ### Harbor的Nginx配置 ```bash # 修改nginx默认的端口监听 [root@k8s99-200 harbor]# vim /etc/nginx/nginx.conf http { # ...注释掉server段的内容 # server { # listen 80 default_server; # listen [::]:80 default_server; # ... # 创建Harbor的代理 [root@k8s99-200 harbor]# vim /etc/nginx/conf.d/harbor.study.com.conf # 添加以下内容 server { listen 80; server_name harbor.study.com; client_max_body_size 1000m; location / { proxy_pass http://127.0.0.1:180; } } ``` `client_max_body_size`：限制请求体的大小，若超过所设定的大小，返回413错误。因为Docker镜像每层可能比较大，如果设置的值太小了，上传就会失败。 ### 测试并启动Nginx ```bash # 测试nginx配置文件是否正确 [root@k8s99-200 harbor]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful # 启动nginx [root@k8s99-200 harbor]# systemctl start nginx # 将nginx设置为开机启动 [root@k8s99-200 harbor]# systemctl enable nginx Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service. ``` ### 99.101上配置DNS解析Harbor的域名 测试`harbor.study.com`域名无法解析 ```bash [root@k8s99-200 harbor]# nslookup harbor.study.com Server: 192.168.99.101 Address: 192.168.99.101#53 ** server can't find harbor.study.com: NXDOMAIN ``` 原因是DNS上没有配置Harbor的A记录，需要修改DNS的服务器 ```bash # bind9配置到192.168.99.101这台主机上的 [root@k8s99-101 ~]# vim /var/named/study.com.zone [root@k8s99-101 ~]# cat /var/named/study.com.zone $ORIGIN study.com. $TTL 600 ; 过期时间10分钟 @ IN SOA dns.study.com. dnsadmin.study.com. ( 2020020201 ; serial 10800 ; refresh（3小时） 900 ; retry（15分钟） 604800 ; expire（1周） 86400 ) ; minimum（1天） NS dns.study.com. $TTL 60 ; 过期时间1分钟 dns A 192.168.99.101 harbor A 192.168.99.200 # 添加完需要重启服务 [root@k8s99-101 ~]# systemctl restart named # 测试解析 [root@k8s99-101 ~]# dig -t A harbor.study.com +short 192.168.99.200 # 同时查询域名也是正确的 [root@k8s99-101 ~]# nslookup harbor.study.com Server: 192.168.99.101 Address: 192.168.99.101#53 Name: harbor.study.com Address: 192.168.99.200 ``` 如上就可以在`/var/named/study.com.zone`业务域中添加`harbor`的A记录，实现Harbor域名访问 ### 本机测试访问Harbor ```bash [root@k8s99-200 harbor]# curl http://harbor.study.com <html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx/1.16.1</center> </body> </html> # 但是直接访问180端口是可以的 [root@k8s99-200 harbor]# curl http://127.0.0.1:180 <!doctype html> <html> <head> <meta charset="utf-8"> <title>Harbor</title> <base href="/"> <meta name="viewport" content="width=device-width, initial-scale=1"> <link rel="icon" type="image/x-icon" href="favicon.ico?v=2"> <link rel="stylesheet" href="styles.701dc5ee3007bd83bfa4.css"></head> <body> <harbor-app> <div class="spinner spinner-lg app-loading"> Loading... </div> </harbor-app> <script src="runtime.9ad22a88fcc70a015907.js" defer></script><script src="polyfills-es5.d01e8ad6bc0c07b49ab6.js" nomodule defer></script><script src="scripts.7fa3fa51e1a86dfba2c8.js" defer></script><script src="main.93d9806e0f1f6ecb9a7f.js" defer></script></body> ``` 以及浏览器访问均会出现`502 Bad Gateway`错误。 ### 本机访问错误处理 #### Nginx域名转发Permission denied 始终是502错误，查看nginx日志 ```bash [root@k8s99-200 ~]# tail -f /var/log/nginx/error.log 2020/03/20 18:05:42 [crit] 25974#0: *3 connect() to 127.0.0.1:180 failed (13: Permission denied) while connecting to upstream, client: 127.0.0.1, server: harbor.study.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:180/", host: "127.0.0.1" ``` 这种一般与防火墙无关，查阅资料后，有两种方案 - 关闭SELinux：`getenforce`->`setenforce 0`->`getenforce`，永久关闭`sed -i 's/\(^SELINUX=\).*/\SELINUX=disabled/' /etc/selinux/config` - 执行`setsebool -P httpd_can_network_connect 1` 这儿我选择了第二种方法 ```bash [root@k8s99-200 harbor]# setsebool -P httpd_can_network_connect 1 # 可以看到域名能正常访问了 [root@k8s99-200 harbor]# curl http://harbor.study.com <!doctype html> <html> <head> <meta charset="utf-8"> <title>Harbor</title> <base href="/"> <meta name="viewport" content="width=device-width, initial-scale=1"> <link rel="icon" type="image/x-icon" href="favicon.ico?v=2"> <link rel="stylesheet" href="styles.701dc5ee3007bd83bfa4.css"></head> <body> <harbor-app> <div class="spinner spinner-lg app-loading"> Loading... </div> </harbor-app> <script src="runtime.9ad22a88fcc70a015907.js" defer></script><script src="polyfills-es5.d01e8ad6bc0c07b49ab6.js" nomodule defer></script><script src="scripts.7fa3fa51e1a86dfba2c8.js" defer></script><script src="main.93d9806e0f1f6ecb9a7f.js" defer></script></body> ``` ### 防火墙允许http服务 首先运维管理主机`192.168.99.200`防火墙允许访问http，外部才能访问到 ```bash [root@k8s99-200 harbor]# firewall-cmd --zone=public --add-service=http success [root@k8s99-200 harbor]# firewall-cmd --zone=public --add-service=http --permanent success [root@k8s99-200 harbor]# firewall-cmd --reload success ``` ### 浏览器访问Harbor 由于在VMware宿主机和虚拟机使用NAT网络连接，需要在NAT网卡（VMnet8）上添加DNS`192.168.99.101`才能解析虚拟机中的域名。 ```bash C:\Users\StarMeow>nslookup harbor.study.com 服务器: UnKnown Address: 192.168.99.101 名称: harbor.study.com Address: 192.168.99.200 ``` 然后浏览器访问 http://harbor.study.com/ 就可以显示Harbor的页面了，默认用户名：admin、密码：Harbor12345 ### 浏览器访问错误处理 #### 浏览器访问Connection refused ```bash [root@k8s99-200 ~]# tail -f /var/log/nginx/error.log 2020/04/20 23:22:56 [error] 7378#0: *444 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.99.1, server: harbor.study.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:180/favicon.ico", host: "harbor.study.com", referrer: "http://harbor.study.com/" ``` 这种一般是Harbor服务没有正常开启。 #### 浏览器访问Connection reset by peer ```bash [root@k8s99-200 ~]# tail -f /var/log/nginx/error.log 2020/04/20 23:19:43 [error] 7378#0: *438 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.99.1, server: harbor.study.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:180/favicon.ico", host: "harbor.study.com", referrer: "http://harbor.study.com/" ``` 尝试直接本机连接Harbor也出现了 ```bash [root@k8s99-200 ~]# curl http://127.0.0.1:180 curl: (7) Failed connect to 127.0.0.1:180; 拒绝连接 ``` 由于是虚拟机运行，直接是暂停的虚拟机，恢复后，虽然Harbor所依赖的所有容器都是正常运行的，但也出现这个错误。之后重启了Harbor所在的服务器，再执行启动Harbor `docker-compose start`就好了 ```bash [root@k8s99-200 ~]# cd /opt/harbor [root@k8s99-200 harbor]# ls common common.sh docker-compose.yml harbor.v1.10.1.tar.gz harbor.yml install.sh LICENSE prepare # 查看机器重启后，容器并没有完全启动 [root@k8s99-200 harbor]# docker-compose ps Name Command State Ports ----------------------------------------------------------------------------------------- harbor-core /harbor/harbor_core Exit 255 harbor-db /docker-entrypoint.sh Exit 255 harbor-jobservice /harbor/harbor_jobservice ... Exit 255 harbor-log /bin/sh -c /usr/local/bin/ ... Up 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Exit 255 nginx nginx -g daemon off; Exit 255 redis redis-server /etc/redis.conf Exit 255 registry /home/harbor/entrypoint.sh Up 5000/tcp registryctl /home/harbor/start.sh Exit 255 [root@k8s99-200 harbor]# docker-compose start Starting log ... done Starting registry ... done Starting registryctl ... done Starting postgresql ... done Starting portal ... done Starting redis ... done Starting core ... done Starting jobservice ... done Starting proxy ... done ``` ### Harbor新建项目测试push 访问Harbor页面，点击新建项目`public`，访问级别选择公开。 ```bash # 下载nginx镜像 [root@k8s99-200 harbor]# docker pull nginx [root@k8s99-200 harbor]# docker tag nginx:latest harbor.study.com/public/nginx:latest # 推送镜像失败 [root@k8s99-200 harbor]# docker push harbor.study.com/public/nginx:latest The push refers to repository [harbor.study.com/public/nginx] 55a77731ed26: Preparing 71f2244bc14d: Preparing f2cb0ecef392: Preparing denied: requested access to the resource is denied # 原因是没有登录 [root@k8s99-200 harbor]# docker login harbor.study.com Username: admin Password: # Harbor12345 WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded # 登陆后再次push就可以了 [root@k8s99-200 harbor]# docker push harbor.study.com/public/nginx:latest The push refers to repository [harbor.study.com/public/nginx] 55a77731ed26: Pushed 71f2244bc14d: Pushed f2cb0ecef392: Pushed latest: digest: sha256:3936fb3946790d711a68c58be93628e43cbca72439079e16d154b5db216b58da size: 948 ``` 在 http://harbor.study.com/harbor/projects 中项目---public---镜像仓库中就可以看到才推送的镜像。 ### 设置Harbor开机启动 将运行脚本添加到开机脚本中，即进入`/opt/harbor`目录，使用`docker-compose start`启动Harbor相关容器。 ```bash [root@k8s99-200 harbor]# echo "cd /opt/harbor && docker-compose ps && docker-compose start" >> /etc/rc.d/rc.local [root@k8s99-200 harbor]# cat /etc/rc.d/rc.local ```

明人不说暗话 领红包就是支持！！！ 来都来了，再到页面底部看下呗~ ↓ ↓ ↓

本文地址： http://blog.starmeow.cn/detail/7f100b7b36092fb9b06dfb4fac360931/

很赞哦！ (1)