Apache Doris 发布流程

Apache 的发布必须至少是 IPMC 成员，拥有 apache 邮箱的commiter，这个角色叫做 release manager。

发布的大致流程如下：

  1. 环境准备
  2. 发布准备
    1. 社区发起 DISCUSS 并与社区交流具体发布计划
    2. 创建分支用于发布
    3. 清理 issue
    4. 将必要的 Patch 合并到发布的分支
  3. 社区发布投票流程
    1. 将 tag 打包，签名并上传到Apache Dev svn 仓库Apache Doris 发布流程 - 图1
    2. 在 Doris 社区发起投票
    3. 投票通过后，在Doris社区发 Result 邮件
    4. 在 Incubator 社区 发起新一轮投票
    5. 发 Result 邮件到 general@incubator.apache.org
  4. 完成工作
    1. 上传签名的软件包到 Apache release repoApache Doris 发布流程 - 图2，并生成相关链接
    2. 准备 release note 并发 Announce 邮件到 general@incubator.apache.org
    3. 在 Doris 官网和 github 发布下载链接

准备环境

如果这是你第一次发布，那么你需要在你的环境中准备如下工具

  1. release signing https://www.apache.org/dev/release-signing.html
  2. gpg https://www.apache.org/dev/openpgp.html
  3. svn https://www.apache.org/dev/openpgp.html

准备gpg key

Release manager 在发布前需要先生成自己的签名公钥，并上传到公钥服务器，之后就可以用这个公钥对准备发布的软件包进行签名。 如果在KEYApache Doris 发布流程 - 图3里已经存在了你的KEY，那么你可以跳过这个步骤了。

签名软件 GnuPG 的安装配置

GnuPG

1991年，程序员 Phil Zimmermann 为了避开政府监视，开发了加密软件PGP。这个软件非常好用，迅速流传开来，成了许多程序员的必备工具。但是，它是商业软件，不能自由使用。所以，自由软件基金会决定，开发一个PGP的替代品，取名为GnuPG。这就是GPG的由来。

安装配置

CentOS 安装命令：

  1. yum install gnupg

安装完成后，默认配置文件 gpg.conf 会放在 home 目录下。

  1. ~/.gnupg/gpg.conf

如果不存在这个目录或文件，可以直接创建一个空文件。 编辑gpg.conf, 修改或者增加 keyserver 配置：

  1. keyserver hkp://keys.gnupg.net

Apache 签名推荐 SHA512， 可以通过配置 gpg 完成。 编辑gpg.conf, 增加下面的三行：

  1. personal-digest-preferences SHA512
  2. cert-digest-algo SHA512
  3. default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed

生成新的签名

准备签名

推荐的生成新签名的设置：

这里必须通过 SecureCRT 等终端直接登录用户账户，不能通过 su - user 或者 ssh 转，否则密码输入 box 会显示不出来而报错。

先看下 gpg 的 version 以及是否支持 SHA512.

  1. $ gpg --version
  2. gpg (GnuPG) 2.0.22
  3. libgcrypt 1.5.3
  4. Copyright (C) 2013 Free Software Foundation, Inc.
  5. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  6. This is free software: you are free to change and redistribute it.
  7. There is NO WARRANTY, to the extent permitted by law.
  9. Home: ~/.gnupg
  10. Supported algorithms:
  11. Pubkey: RSA, ?, ?, ELG, DSA
  12. Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
  13.         CAMELLIA128, CAMELLIA192, CAMELLIA256
  14. Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
  15. Compression: Uncompressed, ZIP, ZLIB, BZIP2
生成新的签名
  1. $ gpg --gen-key
  2. gpg (GnuPG) 2.0.22; Copyright (C) 2013 Free Software Foundation, Inc.
  3. This is free software: you are free to change and redistribute it.
  4. There is NO WARRANTY, to the extent permitted by law.
  6. Please select what kind of key you want:
  7.    (1) RSA and RSA (default)
  8.    (2) DSA and Elgamal
  9.    (3) DSA (sign only)
  10.    (4) RSA (sign only)
  11. Your selection? 1
  12. RSA keys may be between 1024 and 4096 bits long.
  13. What keysize do you want? (2048) 4096
  14. Requested keysize is 4096 bits
  15. Please specify how long the key should be valid.
  16.          0 = key does not expire
  17.       <n>  = key expires in n days
  18.       <n>w = key expires in n weeks
  19.       <n>m = key expires in n months
  20.       <n>y = key expires in n years
  21. Key is valid for? (0)
  22. Key does not expire at all
  23. Is this correct? (y/N) y
  25. GnuPG needs to construct a user ID to identify your key.
  27. Real name: xxx
  28. Name must be at least 5 characters long
  29. Real name: xxx-yyy
  30. Email address: xxx@apache.org
  31. Comment: xxx's key
  32. You selected this USER-ID:
  33.     "xxx-yyy (xxx's key) <xxx@apache.org>"
  35. Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o

其中 Real name 需保持和 id.apache.org 中显示的 id 一致。 Email address 为 apache 的邮箱。

查看和输出

第一行显示公钥文件名（pubring.gpg），第二行显示公钥特征（4096位，Hash字符串和生成时间），第三行显示”用户ID”，第四行显示私钥特征。

  1. $ gpg --list-keys
  2. /home/lide/.gnupg/pubring.gpg
  3. -----------------------------
  4. pub   4096R/33DBF2E0 2018-12-06
  5. uid                  xxx-yyy  (xxx's key) <xxx@apache.org>
  6. sub   4096R/0E8182E6 2018-12-06

其中 xxx-yyy 就是用户ID。

  1. gpg --armor --output public-key.txt --export [用户ID]
  1. $ gpg --armor --output public-key.txt --export xxx-yyy
  2. $ cat public-key.txt
  3. -----BEGIN PGP PUBLIC KEY BLOCK-----
  4. Version: GnuPG v2.0.22 (GNU/Linux)
  6. mQINBFwJEQ0BEACwqLluHfjBqD/RWZ4uoYxNYHlIzZvbvxAlwS2mn53BirLIU/G3
  7. 9opMWNplvmK+3+gNlRlFpiZ7EvHsF/YJOAP59HmI2Z...

上传签名公钥

公钥服务器是网络上专门储存用户公钥的服务器。send-keys 参数可以将公钥上传到服务器。

  1. gpg --send-keys xxxx

其中 xxxx 为上一步 --list-keys 结果中 pub 后面的字符串，如上为：33DBF2E0

也可以通过网站Apache Doris 发布流程 - 图4上传上述 public-key.txt 的内容：

上传成功之后，可以通过查询这个网站Apache Doris 发布流程 - 图5，输入 0x33DBF2E0 查询：

该网站查询有延迟，可能需要等1个小时。

生成 fingerprint 并上传到 apache 用户信息中

由于公钥服务器没有检查机制，任何人都可以用你的名义上传公钥，所以没有办法保证服务器上的公钥的可靠性。通常，你可以在网站上公布一个公钥指纹，让其他人核对下载到的公钥是否为真。

fingerprint参数生成公钥指纹：

  1. gpg --fingerprint [用户ID]
  1. $ gpg --fingerprint xxx-yyy
  2. pub   4096R/33DBF2E0 2018-12-06
  3.       Key fingerprint = 07AA E690 B01D 1A4B 469B  0BEF 5E29 CE39 33DB F2E0
  4. uid                  xxx-yyy (xxx's key) <xxx@apache.org>
  5. sub   4096R/0E8182E6 2018-12-06

将上面的 fingerprint （即 07AA E690 B01D 1A4B 469B 0BEF 5E29 CE39 33DB F2E0）粘贴到自己的用户信息中：

https://id.apache.org

OpenPGP Public Key Primary Fingerprint:

生成 keys

  1. svn co //dist.apache.org/repos/dist/dev/incubator/doris/
  2. # edit doris/KEY file
  3. gpg --list-sigs [用户 ID] >> doris/KEYS
  4. gpg --armor --export [用户 ID] >> doris/KEYS
  5. svn ci --username $ASF_USERNAME --password "$ASF_PASSWORD" -m"Update KEYS"

准备发布

在社区发起 DISCUSS

如果觉得已经修复了很多bug，开发了比较重要的 feature，任何 IPMC 成员都可以发起 DISCUSS 讨论发布新版本。 可以发起一个标题为 [DISCUSS] x.y.z release 的邮件，在社区内部进行讨论，说明已经修复了哪些bug，开发了哪些 features。 如果 DISCUSS 邮件得到大家支持就可以进行下一步。

准备分支

发布前需要先新建一个分支。例如：

  1. $ git checkout -b branch-0.9

这个分支要进行比较充分的测试，使得功能可用，bug收敛，重要bug都得到修复。 这个过程需要等待社区，看看是否有必要的patch需要在这个版本合入，如果有，需要把它 cherry-pick 到发布分支。

清理issue

将属于这个版本的所有 issue 都过一遍，关闭已经完成的，如果没法完成的，推迟到更晚的版本。

合并必要的Patch

在发布等待过程中，可能会有比较重要的Patch合入，如果社区有人说要有重要的Bug需要合入，那么 Release Manager 需要评估并将重要的Patch合入到发布分支中。

社区发布投票流程

打 tag

当上述分支已经比较稳定后，就可以在此分支上打 tag。 记得在创建 tag 时，修改 gensrc/script/gen_build_version.sh 中的 build_version 变量。如 build_version="0.10.0-release"

例如：

  1. $ git checkout branch-0.9
  2. $ git tag -a 0.9.0-rc01 -m "0.9.0 release candidate 01"
  3. $ git push origin 0.9.0-rc01
  4. Counting objects: 1, done.
  5. Writing objects: 100% (1/1), 165 bytes | 0 bytes/s, done.
  6. Total 1 (delta 0), reused 0 (delta 0)
  7. To git@github.com:apache/incubator-doris.git
  8.  * [new tag]         0.9.0-rc01 -> 0.9.0-rc01
  10. $ git tag

打包、签名上传

如下步骤，也需要通过 SecureCRT 等终端直接登录用户账户，不能通过 su - user 或者 ssh 转，否则密码输入 box 会显示不出来而报错。

  1. $ git checkout 0.9.0-rc01
  3. $ git archive --format=tar 0.9.0-rc01 --prefix=apache-doris-0.9.0-incubating-src/ | gzip > apache-doris-0.9.0-incubating-src.tar.gz
  5. $ gpg -u xxx@apache.org --armor --output apache-doris-0.9.0-incubating-src.tar.gz.asc --detach-sign apache-doris-0.9.0-incubating-src.tar.gz
  7. $ gpg --verify apache-doris-0.9.0-incubating-src.tar.gz.asc apache-doris-0.9.0-incubating-src.tar.gz
  9. $ sha512sum apache-doris-0.9.0-incubating-src.tar.gz > apache-doris-0.9.0-incubating-src.tar.gz.sha512
  11. $ sha512sum --check apache-doris-0.9.0-incubating-src.tar.gz.sha512

然后将打包的内容上传到svn仓库中，首先下载 svn 库：

  1. svn co https://dist.apache.org/repos/dist/dev/incubator/doris/

将之前得到的全部文件组织成以下svn路径

  1. ./doris/
  2. |-- 0.11.0-rc1
  3. |   |-- apache-doris-0.11.0-incubating-src.tar.gz
  4. |   |-- apache-doris-0.11.0-incubating-src.tar.gz.asc
  5. |   `-- apache-doris-0.11.0-incubating-src.tar.gz.sha512
  6. `-- KEYS

上传这些文件

  1. svn add 0.11.0-rc1
  2. svn commit -m "Add 0.11.0-rc1"

发社区投票邮件

[VOTE] Release Apache Doris 0.9.0-incubating-rc01

  1. Hi all,
  3. Please review and vote on Apache Doris 0.9.0-incubating-rc01 release.
  5. The release candidate has been tagged in GitHub as 0.9.0-rc01, available
  6. here:
  7. https://github.com/apache/incubator-doris/releases/tag/0.9.0-rc01
  9. Release Notes are here:
  10. https://github.com/apache/incubator-doris/issues/1891
  12. Thanks to everyone who has contributed to this release.
  14. The artifacts (source, signature and checksum) corresponding to this release
  15. candidate can be found here:
  16. https://dist.apache.org/repos/dist/dev/incubator/doris/0.9/0.9.0-rc1/
  18. This has been signed with PGP key 33DBF2E0, corresponding to
  19. lide@apache.org.
  20. KEYS file is available here:
  21. https://dist.apache.org/repos/dist/dev/incubator/doris/KEYS
  22. It is also listed here:
  23. https://people.apache.org/keys/committer/lide.asc
  25. To verify and build, you can refer to following wiki:
  26. https://github.com/apache/incubator-doris/wiki/How-to-verify-Apache-Release
  27. https://wiki.apache.org/incubator/IncubatorReleaseChecklist
  29. The vote will be open for at least 72 hours.
  30. [ ] +1 Approve the release
  31. [ ] +0 No opinion
  32. [ ] -1 Do not release this package because ...
  34. Best Regards,
  35. xxx
  37. ----
  38. DISCLAIMER-WIP: 
  39. Apache Doris is an effort undergoing incubation at The Apache Software Foundation (ASF), 
  40. sponsored by the Apache Incubator. Incubation is required of all newly accepted projects 
  41. until a further review indicates that the infrastructure, communications, and decision 
  42. making process have stabilized in a manner consistent with other successful ASF projects. 
  43. While incubation status is not necessarily a reflection of the completeness or stability 
  44. of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
  46. Some of the incubating project’s releases may not be fully compliant with ASF policy. For 
  47. example, releases may have incomplete or un-reviewed licensing conditions. What follows is 
  48. a list of known issues the project is currently aware of (note that this list, by definition, 
  49. is likely to be incomplete): 
  51.  * Releases may have incomplete licensing conditions
  53. If you are planning to incorporate this work into your product/project, please be aware that
  54. you will need to conduct a thorough licensing review to determine the overall implications of 
  55. including this work. For the current status of this project through the Apache Incubator 
  56. visit: https://incubator.apache.org/projects/doris.html

投票通过后，发 Result 邮件

[Result][VOTE] Release Apache Doris 0.9.0-incubating-rc01

  1. Thanks to everyone, and this vote is now closed.
  3. It has passed with 4 +1 (binding) votes and no 0 or -1 votes.
  5. Binding:
  6. +1 Zhao Chun
  7. +1 xxx
  8. +1 Li Chaoyong
  9. +1 Mingyu Chen
  11. Best Regards,
  12. xxx

发邮件到 general@incubator.apache.org 进行投票

[VOTE] Release Apache Doris 0.9.0-incubating-rc01

  1. Hi all,
  3. Please review and vote on Apache Doris 0.9.0-incubating-rc01 release.
  5. Apache Doris is an MPP-based interactive SQL data warehousing for reporting and analysis.
  7. The Apache Doris community has voted on and approved this release:
  8. https://lists.apache.org/thread.html/d70f7c8a8ae448bf6680a15914646005c6483564464cfa15f4ddc2fc@%3Cdev.doris.apache.org%3E
  10. The vote result email thread:
  11. https://lists.apache.org/thread.html/64d229f0ba15d66adc83306bc8d7b7ccd5910ecb7e842718ce6a61da@%3Cdev.doris.apache.org%3E
  13. The release candidate has been tagged in GitHub as 0.9.0-rc01, available here:
  14. https://github.com/apache/incubator-doris/releases/tag/0.9.0-rc01
  16. There is no CHANGE LOG file because this is the first release of Apache Doris.
  17. Thanks to everyone who has contributed to this release, and there is a simple release notes can be found here:
  18. https://github.com/apache/incubator-doris/issues/406
  20. The artifacts (source, signature and checksum) corresponding to this release candidate can be found here:
  21. https://dist.apache.org/repos/dist/dev/incubator/doris/0.9/0.9.0-rc01/
  23. This has been signed with PGP key 33DBF2E0, corresponding to lide@apache.org.
  24. KEYS file is available here:
  25. https://dist.apache.org/repos/dist/dev/incubator/doris/KEYS
  26. It is also listed here:
  27. https://people.apache.org/keys/committer/lide.asc
  29. The vote will be open for at least 72 hours.
  30. [ ] +1 Approve the release
  31. [ ] +0 No opinion
  32. [ ] -1 Do not release this package because ...
  34. To verify and build, you can refer to following instruction:
  36. Firstly, you must be install and start docker service, and then you could build Doris as following steps:
  38. Step1: Pull the docker image with Doris building environment
  39. $ docker pull apachedoris/doris-dev:build-env
  40. You can check it by listing images, its size is about 3.28GB.
  42. Step2: Run the Docker image
  43. You can run image directly:
  44. $ docker run -it apachedoris/doris-dev:build-env
  46. Step3: Download Doris source
  47. Now you should in docker environment, and you can download Doris source package.
  48. (If you have downloaded source and it is not in image, you can map its path to image in Step2.)
  49. $ wget https://dist.apache.org/repos/dist/dev/incubator/doris/0.9/0.9.0-rc01/apache-doris-0.9.0.rc01-incubating-src.tar.gz
  51. Step4: Build Doris
  52. Now you can decompress and enter Doris source path and build Doris.
  53. $ tar zxvf apache-doris-0.9.0.rc01-incubating-src.tar.gz
  54. $ cd apache-doris-0.9.0.rc01-incubating-src
  55. $ sh build.sh
  57. Best Regards,
  58. xxx
  60. ----
  61. DISCLAIMER-WIP: 
  62. Apache Doris is an effort undergoing incubation at The Apache Software Foundation (ASF), 
  63. sponsored by the Apache Incubator. Incubation is required of all newly accepted projects 
  64. until a further review indicates that the infrastructure, communications, and decision 
  65. making process have stabilized in a manner consistent with other successful ASF projects. 
  66. While incubation status is not necessarily a reflection of the completeness or stability 
  67. of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
  69. Some of the incubating project’s releases may not be fully compliant with ASF policy. For 
  70. example, releases may have incomplete or un-reviewed licensing conditions. What follows is 
  71. a list of known issues the project is currently aware of (note that this list, by definition, 
  72. is likely to be incomplete): 
  74.  * Releases may have incomplete licensing conditions
  76. If you are planning to incorporate this work into your product/project, please be aware that
  77. you will need to conduct a thorough licensing review to determine the overall implications of 
  78. including this work. For the current status of this project through the Apache Incubator 
  79. visit: https://incubator.apache.org/projects/doris.html

邮件的 thread 连接可以在这里找到：

https://lists.apache.org/list.html?dev@doris.apache.org

发 Result 邮件到 general@incubator.apache.org

[RESULT][VOTE] Release Apache Doris 0.9.0-incubating-rc01

  1. Hi,
  3. Thanks to everyone, and the vote for releasing Apache Doris 0.9.0-incubating-rc01 is now closed.
  5. It has passed with 4 +1 (binding) votes and no 0 or -1 votes.
  7. Binding:
  8. +1 Willem Jiang
  9. +1 Justin Mclean
  10. +1 ShaoFeng Shi
  11. +1 Makoto Yui
  13. The vote thread:
  14. https://lists.apache.org/thread.html/da05fdd8d84e35de527f27200b5690d7811a1e97d419d1ea66562130@%3Cgeneral.incubator.apache.org%3E
  16. Best Regards,
  17. xxx

完成发布流程

上传 package 到 release

当正式发布投票成功后，先发[Result]邮件，然后就准备 release package。 将之前在dev下发布的对应rc文件夹下的源码包、签名文件和hash文件拷贝到另一个目录 0.9.0-incubating，注意文件名字中不要rcxx (可以rename，但不要重新计算签名，hash可以重新计算，结果不会变)

第一次发布的话 KEYS 文件也需要拷贝过来。然后add到svn release 下。

  2. https://dist.apache.org/repos/dist/release/incubator/doris/0.9.0-incubating/
  4. 最终能在 apache 官网看到：
  5. http://www.apache.org/dist/incubator/doris/0.9.0-incubating/

发 Announce 邮件到 general@incubator.apache.org

Title:

  1. [ANNOUNCE] Apache Doris (incubating) 0.9.0 Release

发送邮件组：

  1. general@incubator.apache.org <general@incubator.apache.org>
  2. dev@doris.apache.org <dev@doris.apache.org>

邮件正文：

  1. Hi All,
  3. We are pleased to announce the release of Apache Doris 0.9.0-incubating.
  5. Apache Doris (incubating) is an MPP-based interactive SQL data warehousing for reporting and analysis.
  7. The release is available at:
  8. http://doris.apache.org/downloads.html
  10. Thanks to everyone who has contributed to this release, and the release note can be found here:
  11. https://github.com/apache/incubator-doris/releases
  13. Best Regards,
  15. On behalf of the Doris team,
  16. xxx
  18. ----
  19. DISCLAIMER-WIP: 
  20. Apache Doris is an effort undergoing incubation at The Apache Software Foundation (ASF), 
  21. sponsored by the Apache Incubator. Incubation is required of all newly accepted projects 
  22. until a further review indicates that the infrastructure, communications, and decision 
  23. making process have stabilized in a manner consistent with other successful ASF projects. 
  24. While incubation status is not necessarily a reflection of the completeness or stability 
  25. of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
  27. Some of the incubating project’s releases may not be fully compliant with ASF policy. For 
  28. example, releases may have incomplete or un-reviewed licensing conditions. What follows is 
  29. a list of known issues the project is currently aware of (note that this list, by definition, 
  30. is likely to be incomplete): 
  32.  * Releases may have incomplete licensing conditions
  34. If you are planning to incorporate this work into your product/project, please be aware that
  35. you will need to conduct a thorough licensing review to determine the overall implications of 
  36. including this work. For the current status of this project through the Apache Incubator 
  37. visit: https://incubator.apache.org/projects/doris.html

在 Doris 官网和 github 发布链接

创建下载链接

下载链接： http://www.apache.org/dyn/closer.cgi?filename=incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz&action=download

wget —trust-server-names “https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz“

原始位置: https://www.apache.org/dist/incubator/doris/0.9.0-incubating/

http://www.apache.org/dyn/closer.cgi/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz

源码包（source package）: http://www.apache.org/dyn/closer.cgi/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz

ASC: http://archive.apache.org/dist/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz.asc

sha512: http://archive.apache.org/dist/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz.sha512

KEYS: http://archive.apache.org/dist/incubator/doris/KEYS

refer to: http://www.apache.org/dev/release-download-pages#closerApache Doris 发布流程 - 图6

准备 release note

需要修改如下两个地方：

1、Github 的 release 页面

  1. https://github.com/apache/incubator-doris/releases/tag/0.9.0-rc01

2、Doris 官网下载页面

  1. http://doris.apache.org/downloads.html