Publish of Apache Doris

Apache publishing must be at least an IPMC member, a commiter with Apache mailboxes, a role called release manager.

The general process of publication is as follows:

  1. Launching DISCUSS in the community;
  2. Preparing branches and tagging;
  3. Packing tag for signature;
  4. Upload the signature package to the DEV directory of Apache SVN
  5. Send community voting email
  6. Result e-mail after the vote is passed
  7. Send an email to general@incubator.apache.org for a vote.
  8. Email Result to general@incubator.apache.org
  9. Upload the signature package to the release directory of Apache SVN and generate relevant links
  10. Prepare release note and send Announce mail to general@incubator.apache.org
  11. Publish download links on Doris website and GitHub

Release manager needs Mr. A to sign his own public key before publishing and upload it to the public key server. Then he can use this public key to sign the package ready for publication.

1. Prepare for release

1.1 Launching DISCUSS in the Community

If you think you’ve fixed a lot of bugs and developed more important features, any IPMC member can initiate DISCUSS discussions to release a new version. An e-mail entitled [DISCUSS] x.y.z release can be launched to discuss within the community what bugs have been fixed and what features have been developed. If DISCUSS mail is supported, we can proceed to the next step.

1.2 Preparatory Branch

Before publishing, we need to build a new branch, which needs to be fully tested to make functions available, bug convergence, and important bugs repaired.

For example:

  1. $ git checkout -b branch-0.9

1.3 dozen Tags

When the above branches are stable, tags can be made on them. Remember to modify the build_version variable in gensrc/script/gen_build_version.sh when creating tags. For example, build_version='0.10.0-release'.

For example:

  1. $ git checkout branch-0.9
  2. $ git tag -a 0.9.0-rc01 -m "0.9.0 release candidate 01"
  3. $ git push origin 0.9.0-rc01
  4. Counting objects: 1, done.
  5. Writing objects: 100% (1/1), 165 bytes | 0 bytes/s, done.
  6. Total 1 (delta 0), reused 0 (delta 0)
  7. To git@github.com:apache/incubator-doris.git
  8.  * [new tag]         0.9.0-rc01 -> 0.9.0-rc01
  10. $ git tag

2. Installation and configuration of signature software GnuPG

2.1 GnuPG

In 1991, programmer Phil Zimmermann developed the encryption software PGP to avoid government surveillance. This software is very useful, spread quickly, has become a necessary tool for many programmers. However, it is commercial software and cannot be used freely. So the Free Software Foundation decided to develop a replacement for PGP, called GnuPG. This is the origin of GPG.

2.2 Installation Configuration

CentOS installation command:

  1. yum install gnupg

After installation, the default configuration file gpg.conf will be placed in the home directory.

  1. ~/.gnupg /gpg.conf

If this directory or file does not exist, you can create an empty file directly. Edit gpg.conf, modify or add KeyServer configuration:

  1. keyserver hkp http://keys.gnupg.net

Apache signature recommends SHA512, which can be done by configuring gpg. Edit gpg.conf and add the following three lines:

  1. personal-digest-preferences SHA512
  2. cert -digest -something SHA512
  3. default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed

3. Generating new signatures

3.1 Prepare to Sign

Recommended settings for generating new signatures:

We must log in to user account directly through SecureCRT and other terminals. We can’t transfer it through Su - user or ssh. Otherwise, the password input box will not show up and make an error.

Let’s first look at the version of GPG and whether it supports SHA512.

  1. $ gpg --version
  2. gpg (GnuPG) 2.0.22
  3. libgcrypt 1.5.3
  4. Copyright (C) 2013 Free Software Foundation, Inc.
  5. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  6. This is free software: you are free to change and redistribute it.
  7. There is NO WARRANTY, to the extent permitted by law.
  9. Home: ~/.gnupg
  10. Supported algorithms:
  11. Pubkey: RSA, ?, ?, ELG, DSA
  12. Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
  13.         CAMELLIA128, CAMELLIA192, CAMELLIA256
  14. Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
  15. Compression: Uncompressed, ZIP, ZLIB, BZIP2

3.2 Generating new signatures

  1. $ gpg --gen-key
  2. gpg (GnuPG) 2.0.22; Copyright (C) 2013 Free Software Foundation, Inc.
  3. This is free software: you are free to change and redistribute it.
  4. There is NO WARRANTY, to the extent permitted by law.
  6. Please select what kind of key you want:
  7.    (1) RSA and RSA (default)
  8.    (2) DSA and Elgamal
  9.    (3) DSA (sign only)
  10.    (4) RSA (sign only)
  11. Your selection? 1
  12. RSA keys may be between 1024 and 4096 bits long.
  13. What keysize do you want? (2048) 4096
  14. Requested keysize is 4096 bits
  15. Please specify how long the key should be valid.
  16.          0 = key does not expire
  17.       <n>  = key expires in n days
  18.       <n>w = key expires in n weeks
  19.       <n>m = key expires in n months
  20.       <n>y = key expires in n years
  21. Key is valid for? (0)
  22. Key does not expire at all
  23. Is this correct? (y/N) y
  25. GnuPG needs to construct a user ID to identify your key.
  27. Real name: xxx
  28. Name must be at least 5 characters long
  29. Real name: xxx-yyy
  30. Email address: xxx@apache.org
  31. Comment: xxx's key
  32. You selected this USER-ID:
  33.     "xxx-yyy (xxx's key) <xxx@apache.org>"
  35. Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o

Real name needs to be consistent with the ID shown in ID. apache. org. Email address is apache’s mailbox.

3.3 View and Output

The first line shows the name of the public key file (pubring. gpg), the second line shows the public key characteristics (4096 bits, Hash string and generation time), the third line shows the “user ID”, and the fourth line shows the private key characteristics.

  1. $ gpg --list-keys
  2. /home/lide/.gnupg/pubring.gpg
  3. -----------------------------
  4. pub   4096R/33DBF2E0 2018-12-06
  5. uid                  xxx-yyy  (xxx's key) <xxx@apache.org>
  6. sub   4096R/0E8182E6 2018-12-06

xxx-yy is the user ID.

gpg —armor —output public-key.txt —export [用户ID]

  1. $ gpg --armor --output public-key.txt --export xxx-yyy
  2. $ cat public-key.txt
  3. -----BEGIN PGP PUBLIC KEY BLOCK-----
  4. Version: GnuPG V2.0.22 (GNU /Linux)
  6. mQINBFwJEQ0BEACwqLluHfjBqD/RWZ4uoYxNYHlIzZvbvxAlwS2mn53BirLIU/G3
  7. 9opMWNplvmK+3+gNlRlFpiZ7EvHsF/YJOAP59HmI2Z...

4. Upload signature public key

Public key servers are servers that store users’public keys exclusively on the network. The send-keys parameter uploads the public key to the server.

gpg —send-keys xxxx

Where XXX is the last step — the string after pub in the list-keys result, as shown above: 33DBF2E0

You can also upload the contents of the above public-key.txt through the following website:

  1. http://keys.gnupg.net

After successful upload, you can query the website and enter 0x33DBF2E0:

http://keys.gnupg.net

Queries on the site are delayed and may take an hour.

5. Generate fingerprint and upload it to Apache user information

Because the public key server has no checking mechanism, anyone can upload the public key in your name, so there is no way to guarantee the reliability of the public key on the server. Usually, you can publish a public key fingerprint on the website and let others check whether the downloaded public key is true or not.

Fingerprint parameter generates public key fingerprints:

  1. gpg --fingerprint [用户ID]
  1. $ gpg --fingerprint xxx-yyy
  2. pub   4096R/33DBF2E0 2018-12-06
  3.       Key fingerprint = 07AA E690 B01D 1A4B 469B  0BEF 5E29 CE39 33DB F2E0
  4. uid                  xxx-yyy (xxx's key) <xxx@apache.org>
  5. sub   4096R/0E8182E6 2018-12-06

Paste the fingerprint above (i.e. 07AA E690 B01D 1A4B 469B 0BEF 5E29 CE39 33DB F2E0) into your user information:

https://id.apache.org OpenPGP Public Key Primary Fingerprint:

6. Generating keys

Create a new file named KEYS and write it as follows (without any modification):

  1. This file contains the PGP keys of various developers.
  3. Users: pgp < KEYS
  4. or
  5.        gpg --import KEYS
  7. Developers:
  8.     pgp -kxa <your name> and append it to this file.
  9. or
  10.     (pgpk -ll <your name> && pgpk -xa <your name>) >> this file.
  11. or
  12.     (gpg --list-sigs <your name>
  13.     && gpg --armor --export <your name>) >> this file.

Then the generation adds the signature information to write:

  1. gpg --list-sigs [User ID] >> KEYS

Finally, the public key addition is imported:

  1. gpg --armor --export [User ID] >> KEYS

7. Packing Signature

The following steps also need to log into user accounts directly through terminals such as SecureCRT, and can not be transferred through Su - user or ssh, otherwise the password input box will not show and error will be reported.

  1. $ git checkout 0.9.0-rc01
  3. $ git archive --format=tar 0.9.0-rc01 --prefix=apache-doris-0.9.0-incubating-src/ | gzip > apache-doris-0.9.0-incubating-src.tar.gz
  5. $ gpg -u xxx@apache.org --armor --output apache-doris-0.9.0-incubating-src.tar.gz.asc --detach-sign apache-doris-0.9.0-incubating-src.tar.gz
  7. $ gpg --verify apache-doris-0.9.0-incubating-src.tar.gz.asc apache-doris-0.9.0-incubating-src.tar.gz
  9. $ sha512sum apache-doris-0.9.0-incubating-src.tar.gz > apache-doris-0.9.0-incubating-src.tar.gz.sha512
  11. $ sha512sum --check apache-doris-0.9.0-incubating-src.tar.gz.sha512

8. Upload signature packages and KEYS files to DEV SVN

First, download the SVN library:

  1. svn co https://dist.apache.org/repos/dist/dev/incubator/doris/

Organize all previous files into the following SVN paths

  1. ./doris/
  2. ├── 0.9
  3. │   └── 0.9.0-rc1
  4. │       ├── apache-doris-0.9.0-incubating-src.tar.gz
  5. │       ├── apache-doris-0.9.0-incubating-src.tar.gz.asc
  6. │       ├── apache-doris-0.9.0-incubating-src.tar.gz.sha512
  7. │       └── KEYS

Upload these files

  1. svn add 0.9.0-rc1
  2. svn commit -m "Release Apache Doris (incubating) 0.9.0 rc1"

9. Send community voting emails

[VOTE] Release Apache Doris 0.9.0-incubating-rc01

  1. Hi all,
  3. Please review and vote on Apache Doris 0.9.0-incubating-rc01 release.
  5. The release candidate has been tagged in GitHub as 0.9.0-rc01, available
  6. here:
  7. https://github.com/apache/incubator-doris/releases/tag/0.9.0-rc01
  9. ===== CHANGE LOG =====
  11. New Features:
  12. ....
  14. ======================
  16. Thanks to everyone who has contributed to this release.
  18. The artifacts (source, signature and checksum) corresponding to this release
  19. candidate can be found here:
  20. https://dist.apache.org/repos/dist/dev/incubator/doris/0.9/0.9.0-rc1/
  22. This has been signed with PGP key 33DBF2E0, corresponding to
  23. lide@apache.org.
  24. KEYS file is available here:
  25. https://dist.apache.org/repos/dist/dev/incubator/doris/KEYS
  26. It is also listed here:
  27. https://people.apache.org/keys/committer/lide.asc
  29. To verify and build, you can refer to following wiki:
  30. https://github.com/apache/incubator-doris/wiki/How-to-verify-Apache-Release
  31. https://wiki.apache.org/incubator/IncubatorReleaseChecklist
  33. The vote will be open for at least 72 hours.
  34. [ ] +1 Approve the release
  35. [ ] +0 No opinion
  36. [ ] -1 Do not release this package because ...
  38. Best Regards,
  39. xxx

10. Email Result after the vote is passed

[Result][VOTE] Release Apache Doris 0.9.0-incubating-rc01

  1. Thanks to everyone, and this vote is now closed.
  3. It has passed with 4 +1 (binding) votes and no 0 or -1 votes.
  5. Binding:
  6. Zhao Chun
  7. +1 xxx
  8. + 1 Li Chaoyong
  9. +1 Mingyu Chen
  11. Best Regards,
  12. xxx

11. Send an e-mail to general@incubator.apache.org for a vote.

[VOTE] Release Apache Doris 0.9.0-incubating-rc01

  1. Hi all,
  3. Please review and vote on Apache Doris 0.9.0-incubating-rc01 release.
  5. Apache Doris is an MPP-based interactive SQL data warehousing for reporting and analysis.
  7. The Apache Doris community has voted on and approved this release:
  8. https://lists.apache.org/thread.html/d70f7c8a8ae448bf6680a15914646005c6483564464cfa15f4ddc2fc@%3Cdev.doris.apache.org%3E
  10. The vote result email thread:
  11. https://lists.apache.org/thread.html/64d229f0ba15d66adc83306bc8d7b7ccd5910ecb7e842718ce6a61da@%3Cdev.doris.apache.org%3E
  13. The release candidate has been tagged in GitHub as 0.9.0-rc01, available here:
  14. https://github.com/apache/incubator-doris/releases/tag/0.9.0-rc01
  16. There is no CHANGE LOG file because this is the first release of Apache Doris.
  17. Thanks to everyone who has contributed to this release, and there is a simple release notes can be found here:
  18. https://github.com/apache/incubator-doris/issues/406
  20. The artifacts (source, signature and checksum) corresponding to this release candidate can be found here:
  21. https://dist.apache.org/repos/dist/dev/incubator/doris/0.9/0.9.0-rc01/
  23. This has been signed with PGP key 33DBF2E0, corresponding to lide@apache.org.
  24. KEYS file is available here:
  25. https://dist.apache.org/repos/dist/dev/incubator/doris/KEYS
  26. It is also listed here:
  27. https://people.apache.org/keys/committer/lide.asc
  29. The vote will be open for at least 72 hours.
  30. [ ] +1 Approve the release
  31. [ ] +0 No opinion
  32. [ ] -1 Do not release this package because ...
  34. To verify and build, you can refer to following instruction:
  36. Firstly, you must be install and start docker service, and then you could build Doris as following steps:
  38. Step1: Pull the docker image with Doris building environment
  39. $ docker pull apachedoris/doris-dev:build-env
  40. You can check it by listing images, its size is about 3.28GB.
  42. Step2: Run the Docker image
  43. You can run image directly:
  44. $ docker run -it apachedoris/doris-dev:build-env
  46. Step3: Download Doris source
  47. Now you should in docker environment, and you can download Doris source package.
  48. (If you have downloaded source and it is not in image, you can map its path to image in Step2.)
  49. $ wget https://dist.apache.org/repos/dist/dev/incubator/doris/0.9/0.9.0-rc01/apache-doris-0.9.0.rc01-incubating-src.tar.gz
  51. Step4: Build Doris
  52. Now you can decompress and enter Doris source path and build Doris.
  53. $ tar zxvf apache-doris-0.9.0.rc01-incubating-src.tar.gz
  54. $ cd apache-doris-0.9.0.rc01-incubating-src
  55. $ sh build.sh
  57. Best Regards,
  58. xxx

The threaded connection for mail can be found here:

https://lists.apache.org/list.html?dev@doris.apache.org

12. Email Result to general@incubator.apache.org

[RESULT][VOTE] Release Apache Doris 0.9.0-incubating-rc01

  1. Hi,
  3. Thanks to everyone, and the vote for releasing Apache Doris 0.9.0-incubating-rc01 is now closed.
  5. It has passed with 4 +1 (binding) votes and no 0 or -1 votes.
  7. Binding:
  8. +1 Willem Jiang
  9. +1 Justin Mclean
  10. +1 ShaoFeng Shi
  11. +1 Makoto Yui
  13. The vote thread:
  14. https://lists.apache.org/thread.html/da05fdd8d84e35de527f27200b5690d7811a1e97d419d1ea66562130@%3Cgeneral.incubator.apache.org%3E
  16. Best Regards,
  17. xxx

13. Upload package to release

When the formal voting is successful, email [Result] first, and then prepare the release package. Copy the source package, signature file and hash file from the corresponding RC folder published under dev to another directory 0.9.0-incubating. Note that the file name does not need rcxx (rename, but do not recalculate signatures, hash can recalculate, the results will not change)

KEYS files also need to be copied if they are first released. Then add to SVN release.

  2. https://dist.apache.org/repos/dist/release/incubator/doris/0.9.0-incubating/
  4. Eventually you can see it on apache's website:
  5. http://www.apache.org/dist/incubator/doris/0.9.0-incubating/

14. Send Announce e-mail to general@incubator.apache.org

Title:

  1. [ANNOUNCE] Apache Doris (incubating) 0.9.0 Release

Send mail group:

  1. general@incubator.apache.org <general@incubator.apache.org >
  2. dev@doris.apache.org <dev@doris.apache.org >

Mail text:

  1. Hi All,
  3. We are pleased to announce the release of Apache Doris 0.9.0-incubating.
  5. Apache Doris (incubating) is an MPP-based interactive SQL data warehousing for reporting and analysis.
  7. The release is available at:
  8. http://doris.apache.org/downloads.html
  10. Thanks to everyone who has contributed to this release, and the release note can be found here:
  11. https://github.com/apache/incubator-doris/releases
  13. Best Regards,
  15. On behalf of the Doris team,
  16. xxx

15. Publish links on Doris website and GitHub

15.1 Create Download Links

Download link: http://www.apache.org/dyn/closer.cgi?filename=incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz&action=download

wget —trust-server-names “https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz“

Original location: https://www.apache.org/dist/incubator/doris/0.9.0-incubating/

http://www.apache.org/dyn/closer.cgi/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz

源码包（source package）: http://www.apache.org/dyn/closer.cgi/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz

ASC: http://archive.apache.org/dist/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz.asc

sha512: http://archive.apache.org/dist/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz.sha512

KEYS: http://archive.apache.org /dist /incubator /doris /KEYS

refer to: http://www.apache.org/dev/release-download-pages#closerPublish of Apache Doris - 图1

15.2 Prepare release note

The following two areas need to be modified:

  1. Github’s release page
  1. https://github.com/apache/incubator-doris/releases/tag/0.9.0-rc01
  1. Doris Official Website Download Page
  1. http://doris.apache.org /downloads.html