harbor
Harbor 是一个CNCF基金会托管的开源的可信的云原生docker registry项目，可以用于存储、签名、扫描镜像内容，Harbor 通过添加一些常用的功能如安全性、身份权限管理等来扩展 docker registry 项目，此外还支持在 registry 之间复制镜像，还提供更加高级的安全功能，如用户管理、访问控制和活动审计等，在新版本中还添加了Helm仓库托管的支持。
Harbor最核心的功能就是给 docker registry 添加上一层权限保护的功能，要实现这个功能，就需要我们在使用 docker login、pull、push 等命令的时候进行拦截，先进行一些权限相关的校验，再进行操作，其实这一系列的操作 docker registry v2 就已经为我们提供了支持，v2 集成了一个安全认证的功能，将安全认证暴露给外部服务，让外部服务去实现。
环境准备
linux 3.10.0-957.5.1.el7.x86_64
centos 7.6.1810
配置 2c2g500g
安装
docker 安装
docker 为centos用户提供了三种安装方式，我们选择第一种，也是官网推荐的安装方式
- 移除旧的docker 依赖
sudo yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
- 设置docker 源
sudo yum install -y yum-utils
sudo yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
- 安装docker引擎，默认最新版本
sudo yum install docker-ce docker-ce-cli containerd.io
- 启动
systemctl enable docker
systemctl start docker
- 测试
docker -v
Docker version 19.03.12, build 48a66213fe
docker-compose 安装
- 下载可执行文件
sudo curl -L "https://github.com/docker/compose/releases/download/1.26.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- 修改权限
sudo chmod +x /usr/local/bin/docker-compose
- 测试
docker-compose -v
docker-compose version 1.26.0, build d4451659
harbor安装
- 下载在线安装包
harbor可通过在线和离线安装，由于离线安装包下载失败，所以选择在线安装
wget https://github.com/goharbor/harbor/releases/download/v2.0.0/harbor-online-installer-v2.0.0.tgz
- 解压
tar xvf harbor-online-installer-v2.0.0.tgz
- 修改配置
harbor通过docker-compose 管理，harbor.yml 是compose 配置文件
备份配置文件：cp harbor.yml.tmpl harbor.yml
修改配置：vim harbor.yml
# Configuration file of Harbor
# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: xx.xx.xx.xx # 对外开发访问的 ip或域名
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 80
# 注释掉https,当然你也可以通过设置证书，开放https，建议生产使用https
# https related config
#https:
# https port for harbor, default is 443
# port: 443
# The path of cert and key files for nginx
# certificate: /your/certificate/path
# private_key: /your/private/key/path
# # Uncomment following will enable tls communication between all harbor components
# internal_tls:
# # set enabled to true means internal tls is enabled
# enabled: true
# # put your cert and key files on dir
# dir: /etc/harbor/tls/internal
# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433
# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345
# Harbor DB configuration
database:
# The password for the root user of Harbor DB. Change this before any production use.
password: root123
# The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
max_idle_conns: 50
# The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
# Note: the default number of connections is 100 for postgres.
max_open_conns: 100
# The default data volume
data_volume: /data/harbor
# Harbor Storage settings by default is using /data dir on local filesystem
# Uncomment storage_service setting If you want to using external storage
# storage_service:
# # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
# # of registry's and chart repository's containers. This is usually needed when the user hosts a internal storage with self signed certificate.
# ca_bundle:
# # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
# # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
# filesystem:
# maxthreads: 100
# # set disable to true when you want to disable registry redirect
# redirect:
# disabled: false
# Clair configuration
clair:
# The interval of clair updaters, the unit is hour, set to 0 to disable the updaters.
updaters_interval: 12
# Trivy configuration
trivy:
# ignoreUnfixed The flag to display only fixed vulnerabilities
ignore_unfixed: false
# skipUpdate The flag to enable or disable Trivy DB downloads from GitHub
#
# You might want to enable this flag in test or CI/CD environments to avoid GitHub rate limiting issues.
# If the flag is enabled you have to manually download the `trivy.db` file and mount it in the
# /home/scanner/.cache/trivy/db/trivy.db path.
skip_update: false
#
# insecure The flag to skip verifying registry certificate
insecure: false
# github_token The GitHub access token to download Trivy DB
#
# Trivy DB contains vulnerability information from NVD, Red Hat, and many other upstream vulnerability databases.
# It is downloaded by Trivy from the GitHub release page https://github.com/aquasecurity/trivy-db/releases and cached
# in the local file system (/home/scanner/.cache/trivy/db/trivy.db). In addition, the database contains the update
# timestamp so Trivy can detect whether it should download a newer version from the Internet or use the cached one.
# Currently, the database is updated every 12 hours and published as a new release to GitHub.
#
# Anonymous downloads from GitHub are subject to the limit of 60 requests per hour. Normally such rate limit is enough
# for production operations. If, for any reason, it's not enough, you could increase the rate limit to 5000
# requests per hour by specifying the GitHub access token. For more details on GitHub rate limiting please consult
# https://developer.github.com/v3/#rate-limiting
#
# You can create a GitHub token by following the instuctions in
# https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line
#
# github_token: xxx
jobservice:
# Maximum number of job workers in job service
max_job_workers: 10
notification:
# Maximum retry count for webhook job
webhook_job_max_retry: 10
chart:
# Change the value of absolute_url to enabled can enable absolute url in chart
absolute_url: disabled
# Log configurations
log:
# options are debug, info, warning, error, fatal
level: info
# configs for logs in local storage
local:
# Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
rotate_count: 50
# Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
# If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
# are all valid.
rotate_size: 200M
# The directory on your host that store log
location: /var/log/harbor
# Uncomment following lines to enable external syslog endpoint.
# external_endpoint:
# # protocol used to transmit log to external endpoint, options is tcp or udp
# protocol: tcp
# # The host of external endpoint
# host: localhost
# # Port of external endpoint
# port: 5140
#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version: 2.0.0
# Uncomment external_database if using external database.
# external_database:
# harbor:
# host: harbor_db_host
# port: harbor_db_port
# db_name: harbor_db_name
# username: harbor_db_username
# password: harbor_db_password
# ssl_mode: disable
# max_idle_conns: 2
# max_open_conns: 0
# clair:
# host: clair_db_host
# port: clair_db_port
# db_name: clair_db_name
# username: clair_db_username
# password: clair_db_password
# ssl_mode: disable
# notary_signer:
# host: notary_signer_db_host
# port: notary_signer_db_port
# db_name: notary_signer_db_name
# username: notary_signer_db_username
# password: notary_signer_db_password
# ssl_mode: disable
# notary_server:
# host: notary_server_db_host
# port: notary_server_db_port
# db_name: notary_server_db_name
# username: notary_server_db_username
# password: notary_server_db_password
# ssl_mode: disable
# Uncomment external_redis if using external Redis server
# external_redis:
# host: redis
# port: 6379
# password:
# # db_index 0 is for core, it's unchangeable
# registry_db_index: 1
# jobservice_db_index: 2
# chartmuseum_db_index: 3
# clair_db_index: 4
# trivy_db_index: 5
# idle_timeout_seconds: 30
# Uncomment uaa for trusting the certificate of uaa instance that is hosted via self-signed cert.
# uaa:
# ca_file: /path/to/ca
# Global proxy
# Config http proxy for components, e.g. http://my.proxy.com:3128
# Components doesn't need to connect to each others via http proxy.
# Remove component from `components` array if want disable proxy
# for it. If you want use proxy for replication, MUST enable proxy
# for core and jobservice, and set `http_proxy` and `https_proxy`.
# Add domain to the `no_proxy` field, when you want disable proxy
# for some special registry.
proxy:
http_proxy:
https_proxy:
no_proxy:
components:
- core
- jobservice
- clair
- trivy
- 安装
cd /home/admin/harbor/
./install.sh
脚本执行完成后查看harbor组成 和运行状态
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3c73fc1bdf49 goharbor/nginx-photon:v2.0.0 "nginx -g 'daemon of…" 44 hours ago Up 44 hours (healthy) 0.0.0.0:80->8080/tcp nginx
cc58e0fd2347 goharbor/harbor-jobservice:v2.0.0 "/harbor/entrypoint.…" 44 hours ago Up 44 hours (healthy) harbor-jobservice
7f5929dba6bc goharbor/harbor-core:v2.0.0 "/harbor/entrypoint.…" 44 hours ago Up 44 hours (healthy) harbor-core
001167acd8fc goharbor/harbor-db:v2.0.0 "/docker-entrypoint.…" 44 hours ago Up 44 hours (healthy) 5432/tcp harbor-db
1684303e0782 goharbor/harbor-registryctl:v2.0.0 "/home/harbor/start.…" 44 hours ago Up 44 hours (healthy) registryctl
aa4b212f6595 goharbor/redis-photon:v2.0.0 "redis-server /etc/r…" 44 hours ago Up 44 hours (healthy) 6379/tcp redis
79189cffe12a goharbor/registry-photon:v2.0.0 "/home/harbor/entryp…" 44 hours ago Up 44 hours (healthy) 5000/tcp registry
c1f45a3e3131 goharbor/harbor-portal:v2.0.0 "nginx -g 'daemon of…" 44 hours ago Up 44 hours (healthy) 8080/tcp harbor-portal
4c85974a6c25 goharbor/harbor-log:v2.0.0 "/bin/sh -c /usr/loc…" 44 hours ago Up 44 hours (healthy) 127.0.0.1:1514->10514/tcp harbor-log
docker-compose ps
Name Command State Ports
---------------------------------------------------------------------------------------------
harbor-core /harbor/entrypoint.sh Up (healthy)
harbor-db /docker-entrypoint.sh Up (healthy) 5432/tcp
harbor-jobservice /harbor/entrypoint.sh Up (healthy)
harbor-log /bin/sh -c /usr/local/bin/ ... Up (healthy) 127.0.0.1:1514->10514/tcp
harbor-portal nginx -g daemon off; Up (healthy) 8080/tcp
nginx nginx -g daemon off; Up (healthy) 0.0.0.0:80->8080/tcp
redis redis-server /etc/redis.conf Up (healthy) 6379/tcp
registry /home/harbor/entrypoint.sh Up (healthy) 5000/tcp
registryctl /home/harbor/start.sh Up (healthy)
- 访问 harbor
使用默认用户名密码登录，登录后可修改 admin : Harbor12345
Harbor 组成
Harbor 配置
- 配置认证方式
harbor 支持 数据库,ldap/ad, oidc等认证方式，这里配置ldap的认证方式，如图，选择LDAP，并填写
LDAP相关信息，关于LDAP和AD可考虑内网协议学习LDAP篇之组和OU介绍，配置好后，点击测试，通过即可
注意：选择ldap后，用户认证管理通过ldap控制，不能通过harbor 新增，删除，管理用户组权限
Harbor 功能
- 项目管理
Harbor中的一个项目包含了一个应用程序所需要的所有repositories。在工程创建之前，并不允许推送镜像到Harbor中。Harbor对于project采用基于角色的访问控制。在Harbor中projects有两种类型：- Public: 所有的用户都具有读取public project的权限， 其主要是为了方便你分享一些repositories
- Private: 一个私有project只能被特定用户以适当的权限进行访问
如图：一个项目下，包含这个对项目的镜像仓库管理（项目下的镜像），成员管理（可访问项目的成员），配置管理（白名单，可访问程度等），标签管理（分类表示），机器人账户（通过机器人账户自动pull ， push 镜像）管理等功能
- 系统管理
基础功能都集中在上图，左侧菜单栏提供了系统管理的功能，包括用户管理，组管理, 仓库管理，复制管理，标签，项目定额，审查服务，垃圾清理等功能
- 用户管理
启用ldap后，初次登录后的用户，就会成为系统可展示的用户 - 组管理
接入ldap后，组管理由ldap的域控决定，由于公司暂无现成的域控分类，所以此处不添加，如果使用数据库认证，这里可自行管理组，方便项目中统一成员管理 - 仓库管理
ldap 2.0 不仅支持 多个harbor系统之间的镜像同步，也支持向第三方docker仓库同步镜像，这个仓库可供复制管理选择 - 复制管理
复制管理允许通过一定的策略，向其他仓库同步镜像，比如镜像集群之间的数据同步 - 标签
这个标签是全局标签 - 项目定额
项目定额可以设置项目可存储镜像的大小，默认不限制 - 审查服务
为了镜像安全，审查服务提供了扫描器，扫描镜像是否有漏洞，2.0 以Trivy作为默认扫描器，也可以配置其他扫描器，如clair等，harbor提供prepare脚本支持一键修改
./prepare --help
prepare base dir is set to /home/admin/harbor/harbor
Usage: main.py prepare [OPTIONS]
Options:
--conf TEXT the path of Harbor configuration file
--with-notary the Harbor instance is to be deployed with notary
--with-clair the Harbor instance is to be deployed with clair
--with-trivy the Harbor instance is to be deployed with Trivy
--with-chartmuseum the Harbor instance is to be deployed with chart
repository supporting
--help Show this message and exit.
Clean up the input dir
- 垃圾清理
垃圾清理用来删除指定规则下的镜像
存储
Harbor支持远程挂载存储，以解决本地存储不足的情况，支持NFS，OSS，分布式存储等
docker client 配置
Harbor搭建好之后，就可以通过docker client上传镜像了，docker安装同上
- 配置镜像加速
配置阿里云镜像加速器
vim /etc/docker/daemon.json
{
"data-root": "/data/docker", # docker数据目录，可不修改
"insecure-registries": ["0.0.0.0/0"], # 非安全地址 你的harbor地址 或者0.0.0.0/0 （由于未使用https认证）
"registry-mirrors": ["https://xxxx.mirror.aliyuncs.com"] # 阿里云加速地址 自己找
}
- 重启docker
service docker restart
systemd 系统使用如下命令
systemctl restart docker
上传镜像
- login
docker login -u username http://xx.xx.xx.xx:port
登录私有镜像仓库，如果没有配置上面的非安全地址，会出现401错误
- tag
我们从dockerhub pull一个node镜像到本地，然后对这个镜像打tag，可直接复制此处命令做修改
查看拉下来的官网镜像
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
node latest 37ad18cd8bd1 2 days ago 943MB
对其打 tag，私有需要在tag中加入仓库地址，如下
docker tag node:latest xx.xx.xx.xx/library/zm-node
查看镜像
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
xx.xx.xx.xx/library/zm-node latest 37ad18cd8bd1 2 days ago 943MB
node latest 37ad18cd8bd1 2 days ago 943MB
- 推送
docker push xx.xx.xx.xx/library/zm-node
- 查看镜像
至此私有仓库的搭建和测试完成
问题归纳
过程中在ldap 和 docker 认证问题上耗时较长，最诡异的问题是 login成功了，push有问题 ，提示
unauthorized to access repository
参考资料
harbor官方文档
harbor使用手册
k8s harbor
内网协议学习LDAP篇之组和OU介绍
来源：oschina
链接：https://my.oschina.net/u/4276152/blog/4337080