09. 部署 docker registry

本文档介绍使用 docker 官方的 registry v2 镜像部署私有仓库的步骤,你也可以参考附件文档部署 Harbor 私有仓库(D.部署 Harbor 私有仓库)。

本文档讲解部署一个 TLS 加密、HTTP Basic 认证、用 ceph rgw 做后端存储的私有 docker registry 步骤,如果使用其它类型的后端存储,则可以从 “创建 docker registry” 节开始;

示例两台机器 IP 如下:

  • ceph rgw: 172.27.132.66
  • docker registry: 172.27.132.67

部署 ceph RGW 节点

  1. $ ceph-deploy rgw create 172.27.132.66 # rgw 默认监听7480端口
  2. $

创建测试账号 demo

  1. $ radosgw-admin user create --uid=demo --display-name="ceph rgw demo user"
  2. $

创建 demo 账号的子账号 swift

当前 registry 只支持使用 swift 协议访问 ceph rgw 存储,暂时不支持 s3 协议;

  1. $ radosgw-admin subuser create --uid demo --subuser=demo:swift --access=full --secret=secretkey --key-type=swift
  2. $

创建 demo:swift 子账号的 sercret key

  1. $ radosgw-admin key create --subuser=demo:swift --key-type=swift --gen-secret
  2. {
  3. "user_id": "demo",
  4. "display_name": "ceph rgw demo user",
  5. "email": "",
  6. "suspended": 0,
  7. "max_buckets": 1000,
  8. "auid": 0,
  9. "subusers": [
  10. {
  11. "id": "demo:swift",
  12. "permissions": "full-control"
  13. }
  14. ],
  15. "keys": [
  16. {
  17. "user": "demo",
  18. "access_key": "5Y1B1SIJ2YHKEHO5U36B",
  19. "secret_key": "nrIvtPqUj7pUlccLYPuR3ntVzIa50DToIpe7xFjT"
  20. }
  21. ],
  22. "swift_keys": [
  23. {
  24. "user": "demo:swift",
  25. "secret_key": "ttQcU1O17DFQ4I9xzKqwgUe7WIYYX99zhcIfU9vb"
  26. }
  27. ],
  28. "caps": [],
  29. "op_mask": "read, write, delete",
  30. "default_placement": "",
  31. "placement_tags": [],
  32. "bucket_quota": {
  33. "enabled": false,
  34. "max_size_kb": -1,
  35. "max_objects": -1
  36. },
  37. "user_quota": {
  38. "enabled": false,
  39. "max_size_kb": -1,
  40. "max_objects": -1
  41. },
  42. "temp_url_keys": []
  43. }
  • ttQcU1O17DFQ4I9xzKqwgUe7WIYYX99zhcIfU9vb 为子账号 demo:swift 的 secret key;

创建 docker registry

创建 registry 使用的 x509 证书

  1. $ mkdir -p registry/{auth,certs}
  2. $ cat > registry-csr.json <<EOF
  3. {
  4. "CN": "registry",
  5. "hosts": [
  6. "127.0.0.1",
  7. "172.27.132.67"
  8. ],
  9. "key": {
  10. "algo": "rsa",
  11. "size": 2048
  12. },
  13. "names": [
  14. {
  15. "C": "CN",
  16. "ST": "BeiJing",
  17. "L": "BeiJing",
  18. "O": "k8s",
  19. "OU": "opsnull"
  20. }
  21. ]
  22. }
  23. EOF
  24. $ cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
  25. -ca-key=/etc/kubernetes/cert/ca-key.pem \
  26. -config=/etc/kubernetes/cert/ca-config.json \
  27. -profile=kubernetes registry-csr.json | cfssljson -bare registry
  28. $ cp registry.pem registry-key.pem registry/certs
  29. $
  • 这里复用以前创建的 CA 证书和秘钥文件;
  • hosts 字段指定 registry 的 NodeIP;

创建 HTTP Baisc 认证文件

  1. $ docker run --entrypoint htpasswd registry:2 -Bbn foo foo123 > registry/auth/htpasswd
  2. $ cat registry/auth/htpasswd
  3. foo:$2y$05$iZaM45Jxlcg0DJKXZMggLOibAsHLGybyU.CgU9AHqWcVDyBjiScN.

配置 registry 参数

  1. export RGW_AUTH_URL="http://172.27.132.66:7480/auth/v1"
  2. export RGW_USER="demo:swift"
  3. export RGW_SECRET_KEY="ttQcU1O17DFQ4I9xzKqwgUe7WIYYX99zhcIfU9vb"
  4. cat > config.yml << EOF
  5. # https://docs.docker.com/registry/configuration/#list-of-configuration-options
  6. version: 0.1
  7. log:
  8. level: info
  9. fromatter: text
  10. fields:
  11. service: registry
  12. storage:
  13. cache:
  14. blobdescriptor: inmemory
  15. delete:
  16. enabled: true
  17. swift:
  18. authurl: ${RGW_AUTH_URL}
  19. username: ${RGW_USER}
  20. password: ${RGW_SECRET_KEY}
  21. container: registry
  22. auth:
  23. htpasswd:
  24. realm: basic-realm
  25. path: /auth/htpasswd
  26. http:
  27. addr: 0.0.0.0:8000
  28. headers:
  29. X-Content-Type-Options: [nosniff]
  30. tls:
  31. certificate: /certs/registry.pem
  32. key: /certs/registry-key.pem
  33. health:
  34. storagedriver:
  35. enabled: true
  36. interval: 10s
  37. threshold: 3
  38. EOF
  39. [k8s@zhangjun-k8s-01 cert]$ cp config.yml registry
  40. [k8s@zhangjun-k8s-01 cert]$ scp -r registry 172.27.132.67:/opt/k8s
  • storage.swift 指定后端使用 swfit 接口协议的存储,这里配置的是 ceph rgw 存储参数;
  • auth.htpasswd 指定了 HTTP Basic 认证的 token 文件路径;
  • http.tls 指定了 registry http 服务器的证书和秘钥文件路径;

创建 docker registry:

  1. ssh k8s@172.27.132.67
  2. $ docker run -d -p 8000:8000 --privileged \
  3. -v /opt/k8s/registry/auth/:/auth \
  4. -v /opt/k8s/registry/certs:/certs \
  5. -v /opt/k8s/registry/config.yml:/etc/docker/registry/config.yml \
  6. --name registry registry:2
  • 执行该 docker run 命令的机器 IP 为 172.27.132.67;

向 registry push image

将签署 registry 证书的 CA 证书拷贝到 /etc/docker/certs.d/172.27.132.67:8000 目录下

  1. [k8s@zhangjun-k8s-01 cert]$ sudo mkdir -p /etc/docker/certs.d/172.27.132.67:8000
  2. [k8s@zhangjun-k8s-01 cert]$ sudo cp /etc/kubernetes/cert/ca.pem /etc/docker/certs.d/172.27.132.67:8000/ca.crt

登陆私有 registry:

  1. $ docker login 172.27.132.67:8000
  2. Username: foo
  3. Password:
  4. Login Succeeded

登陆信息被写入 ~/.docker/config.json 文件:

  1. $ cat ~/.docker/config.json
  2. {
  3. "auths": {
  4. "172.27.132.67:8000": {
  5. "auth": "Zm9vOmZvbzEyMw=="
  6. }
  7. }
  8. }

将本地的 image 打上私有 registry 的 tag:

  1. $ docker tag prom/node-exporter:v0.16.0 172.27.132.67:8000/prom/node-exporter:v0.16.0
  2. $ docker images |grep pause
  3. prom/node-exporter:v0.16.0 latest f9d5de079539 2 years ago 239.8 kB
  4. 172.27.132.67:8000/prom/node-exporter:v0.16.0 latest f9d5de079539 2 years ago 239.8 kB

将 image push 到私有 registry:

  1. $ docker push 172.27.132.67:8000/prom/node-exporter:v0.16.0
  2. The push refers to a repository [172.27.132.67:8000/prom/node-exporter:v0.16.0]
  3. 5f70bf18a086: Pushed
  4. e16a89738269: Pushed
  5. latest: digest: sha256:9a6b437e896acad3f5a2a8084625fdd4177b2e7124ee943af642259f2f283359 size: 916

查看 ceph 上是否已经有 push 的 pause 容器文件:

  1. $ rados lspools
  2. rbd
  3. cephfs_data
  4. cephfs_metadata
  5. .rgw.root
  6. k8s
  7. default.rgw.control
  8. default.rgw.meta
  9. default.rgw.log
  10. default.rgw.buckets.index
  11. default.rgw.buckets.data
  12. $ rados --pool default.rgw.buckets.data ls|grep node-exporter
  13. 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_layers/sha256/cdb7590af5f064887f3d6008d46be65e929c74250d747813d85199e04fc70463/link
  14. 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_manifests/revisions/sha256/55302581333c43d540db0e144cf9e7735423117a733cdec27716d87254221086/link
  15. 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_manifests/tags/v0.16.0/current/link
  16. 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_manifests/tags/v0.16.0/index/sha256/55302581333c43d540db0e144cf9e7735423117a733cdec27716d87254221086/link
  17. 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_layers/sha256/224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c/link
  18. 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_layers/sha256/528dda9cf23d0fad80347749d6d06229b9a19903e49b7177d5f4f58736538d4e/link
  19. 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_layers/sha256/188af75e2de0203eac7c6e982feff45f9c340eaac4c7a0f59129712524fa2984/link

私有 registry 的运维操作

查询私有镜像中的 images

  1. $ curl --user foo:foo123 --cacert /etc/docker/certs.d/172.27.132.67\:8000/ca.crt https://172.27.132.67:8000/v2/_catalog
  2. {"repositories":["prom/node-exporter"]}

查询某个镜像的 tags 列表

  1. $ curl --user foo:foo123 --cacert /etc/docker/certs.d/172.27.132.67\:8000/ca.crt https://172.27.132.67:8000/v2/prom/node-exporter/tags/list
  2. {"name":"prom/node-exporter","tags":["v0.16.0"]}

获取 image 或 layer 的 digest

v2/<repoName>/manifests/<tagName> 发 GET 请求,从响应的头部 Docker-Content-Digest 获取 image digest,从响应的 body 的 fsLayers.blobSum 中获取 layDigests;

注意,必须包含请求头:Accept: application/vnd.docker.distribution.manifest.v2+json

  1. $ curl -v -H "Accept: application/vnd.docker.distribution.manifest.v2+json" --user foo:foo123 --cacert /etc/docker/certs.d/172.27.132.67\:8000/ca.crt https://172.27.132.67:8000/v2/prom/node-exporter/manifests/v0.16.0
  2. * About to connect() to 172.27.132.67 port 8000 (#0)
  3. * Trying 172.27.132.67...
  4. * Connected to 172.27.132.67 (172.27.132.67) port 8000 (#0)
  5. * Initializing NSS with certpath: sql:/etc/pki/nssdb
  6. * CAfile: /etc/docker/certs.d/172.27.132.67:8000/ca.crt
  7. CApath: none
  8. * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  9. * Server certificate:
  10. * subject: CN=registry,OU=4Paradigm,O=k8s,L=BeiJing,ST=BeiJing,C=CN
  11. * start date: Jul 05 12:52:00 2018 GMT
  12. * expire date: Jul 02 12:52:00 2028 GMT
  13. * common name: registry
  14. * issuer: CN=kubernetes,OU=4Paradigm,O=k8s,L=BeiJing,ST=BeiJing,C=CN
  15. * Server auth using Basic with user 'foo'
  16. > GET /v2/prom/node-exporter/manifests/v0.16.0 HTTP/1.1
  17. > Authorization: Basic Zm9vOmZvbzEyMw==
  18. > User-Agent: curl/7.29.0
  19. > Host: 172.27.132.67:8000
  20. > Accept: application/vnd.docker.distribution.manifest.v2+json
  21. >
  22. < HTTP/1.1 200 OK
  23. < Content-Length: 949
  24. < Content-Type: application/vnd.docker.distribution.manifest.v2+json
  25. < Docker-Content-Digest: sha256:55302581333c43d540db0e144cf9e7735423117a733cdec27716d87254221086
  26. < Docker-Distribution-Api-Version: registry/2.0
  27. < Etag: "sha256:55302581333c43d540db0e144cf9e7735423117a733cdec27716d87254221086"
  28. < X-Content-Type-Options: nosniff
  29. < Date: Fri, 06 Jul 2018 06:18:41 GMT
  30. <
  31. {
  32. "schemaVersion": 2,
  33. "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  34. "config": {
  35. "mediaType": "application/vnd.docker.container.image.v1+json",
  36. "size": 3511,
  37. "digest": "sha256:188af75e2de0203eac7c6e982feff45f9c340eaac4c7a0f59129712524fa2984"
  38. },
  39. "layers": [
  40. {
  41. "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
  42. "size": 2392417,
  43. "digest": "sha256:224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c"
  44. },
  45. {
  46. "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
  47. "size": 560703,
  48. "digest": "sha256:cdb7590af5f064887f3d6008d46be65e929c74250d747813d85199e04fc70463"
  49. },
  50. {
  51. "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
  52. "size": 5332460,
  53. "digest": "sha256:528dda9cf23d0fad80347749d6d06229b9a19903e49b7177d5f4f58736538d4e"
  54. }
  55. ]

删除 image

/v2/<name>/manifests/<reference> 发送 DELETE 请求,reference 为上一步返回的 Docker-Content-Digest 字段内容:

  1. $ curl -X DELETE --user foo:foo123 --cacert /etc/docker/certs.d/172.27.132.67\:8000/ca.crt https://172.27.132.67:8000/v2/prom/node-exporter/manifests/sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5
  2. $

删除 layer

/v2/<name>/blobs/<digest>发送 DELETE 请求,其中 digest 是上一步返回的 fsLayers.blobSum 字段内容:

  1. $ curl -X DELETE --user foo:foo123 --cacert /etc/docker/certs.d/172.27.132.67\:8000/ca.crt https://172.27.132.67:8000/v2/prom/node-exporter/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
  2. $ curl -X DELETE --cacert /etc/docker/certs.d/172.27.132.67\:8000/ca.crt https://172.27.132.67:8000/v2/prom/node-exporter/blobs/sha256:04176c8b224aa0eb9942af765f66dae866f436e75acef028fe44b8a98e045515
  3. $

常见问题

login 失败 416

执行 http://docs.ceph.com/docs/master/install/install-ceph-gateway/ 里面的 s3 test.py 程序失败:

[k8s@zhangjun-k8s-01 cert]$ python s3test.py Traceback (most recent call last): File “s3test.py”, line 12, in bucket = conn.create_bucket(‘my-new-bucket’) File “/usr/lib/python2.7/site-packages/boto/s3/connection.py”, line 625, in create_bucket response.status, response.reason, body) boto.exception.S3ResponseError: S3ResponseError: 416 Requested Range Not Satisfiable

解决版办法:

  1. 在管理节点上修改 ceph.conf
  2. ceph-deploy config push zhangjun-k8s-01 zhangjun-k8s-02 zhangjun-k8s-03
  3. systemctl restart ‘ceph-mds@zhangjun-k8s-03.service’ systemctl restart ceph-osd@0 systemctl restart ‘ceph-mon@zhangjun-k8s-01.service’ systemctl restart ‘ceph-mgr@zhangjun-k8s-01.service’

For anyone who is hitting this issue set default pg_num and pgp_num to lower value(8 for example), or set mon_max_pg_per_osd to a high value in ceph.conf radosgw-admin doesn’ throw proper error when internal pool creation fails, hence the upper level error which is very confusing.

https://tracker.ceph.com/issues/21497

login 失败 503

[root@zhangjun-k8s-01 ~]# docker login 172.27.132.67:8000 Username: foo Password: Error response from daemon: login attempt to https://172.27.132.67:8000/v2/ failed with status: 503 Service Unavailable

原因: docker run 缺少 —privileged 参数;