Releases Doris Connectors

Doris Connectors currently contains:

  • Doris Flink Connector
  • Doris Spark Connector

The code base is separate from the main Doris code base and is located at:

Preparing for release

First, see the release preparation documentation to prepare for the release. ## Releasing to Maven

Let’s take the example of releasing Flink Connector v1.0.0.

1. Prepare the branch

Create a branch in the codebase: branch-1.0, and checkout to that branch.

2. release to Maven staging

Since Flink Connector releases different releases for different Flink versions (e.g. 1.11, 1.12, 1.13), we need to handle each version separately.

Let’s take Flink version 1.13.5 and scala version 2.12 as an example.

First, replace flink.version and scala.version in pom.xml with

  1. cd flink-doris-connector/
  2. sed -i 's/\${flink.version}/1.13.5/g' pom.xml
  3. sed -i 's/\${scala.version}/2.12/g' pom.xml

After replacing, commit the local changes to.

  1. git add . -u
  2. git commit -m "prepare for 1.13.5-2.12-1.0.0"

Execute the following command to start generating the release tag.

  1. cd flink-doris-connector/
  2. mvn release:clean -DreleaseArgs="-Dflink.version=1.13.5 -Dscala.version=2.12" -Dflink.version=1.13.5 -Dscala.version=2.12
  3. mvn release:prepare -DreleaseArgs="-Dflink.version=1.13.5 -Dscala.version=2.12" -Dflink.version=1.13.5 -Dscala.version=2.12 -DpushChanges=false

where -DpushChanges=false means that the newly generated branches and tags are not pushed to the codebase during execution.

After executing the release:prepare command, the following three pieces of information will be requested.

  1. the version of the Doris Flink Connector, which we can do by default, either by entering a carriage return or by typing in the version you want. The version format is {flink.version}-{scala.version}-{connector.version}, e.g. 1.13.5-2.12-1.0.0. 2.
  2. The release tag of Doris Flink Connector, the release process will generate a tag locally, we can use the default tag name, such as 1.13.5-2.12-1.0.0.
  3. The version number of the next version of Doris Flink Connector. This version number is only used for generating local branches and has no real meaning. For example, if the current release is 1.13.5-2.12-1.0.0, then the next version number should be 1.13.5-2.12-1.0.1.

mvn release:prepare may ask for GPG passphrase, if you get gpg: no valid OpenPGP data found error, you can try after executing export GPG_TTY=$(tty).

If mvn release:prepare succeeds, a tag and a branch will be created locally, and two new commits will be added to the current branch, the first one corresponding to the newly created tag and the second one to the branch of the next release, which can be viewed via git log.

Once the local tag is verified, you need to push the tag to the repository.

git push upstream --tags

where upstream points to the apache/incubator-doris-flink-connector repository.

Finally, execute perform:

  1. mvn release:perform -DreleaseArgs="-Dflink.version=1.13.5 -Dscala.version=2.12" -Dflink.version=1.13.5 -Dscala.version=2.12

After successful execution, the version just released can be found in https://repository.apache.org/#stagingRepositoriesRelease Doris Connectors - 图1 (opens new window)

Release Doris Connectors - 图2

Note that the .asc signature file needs to be included.

If there is an error. You need to delete the local tag, the tag in the codebase, and the two newly generated local commits. And drop the staging. Then re-execute the above steps.

After checking, click the close button in the figure to finish staging release.

3. Prepare svn

Check out the svn repository.

  1. svn co https://dist.apache.org/repos/dist/dev/incubator/doris/

Package the tag source code and generate the signature file and sha256 checksum file. Here we take 1.13.5-2.12-1.0.0 as an example.

  1. git archive --format=tar 1.13.5-2.12-1.0.0 --prefix=apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src/ | gzip > apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz
  2. gpg -u xxx@apache.org --armor --output apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz.asc  --detach-sign apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz
  3. sha512sum apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz > apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz.sha512

The end result is three files:

  1. apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz
  2. apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz.asc
  3. apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz.sha512

Move these three files to the svn directory:

  1. doris/flink-connector/1.0.0/

The final svn directory structure will look like this:

  1. |____0.15
  2. | |____0.15.0-rc04
  3. | | |____apache-doris-0.15.0-incubating-src.tar.gz.sha512
  4. | | |____apache-doris-0.15.0-incubating-src.tar.gz.asc
  5. | | |____apache-doris-0.15.0-incubating-src.tar.gz
  6. |____KEYS
  7. |____flink-connector
  8. | |____1.0.0
  9. | | |____apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz
  10. | | |____apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz.sha512
  11. | | |____apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz.asc
  12. | | |____apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz.asc
  13. | | |____apache-doris-flink-connector-1.12.7-2.12-1.0.0-incubating-src.tar.gz
  14. | | |____apache-doris-flink-connector-1.12.7-2.12-1.0.0-incubating-src.tar.gz.asc
  15. | | |____apache-doris-flink-connector-1.12.7-2.12-1.0.0-incubating-src.tar.gz.sha512
  16. | | |____apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz
  17. | | |____apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz.sha512

Where 0.15 is the directory of Doris main code, and under flink-connector/1.0.0 is the content of this release.

Note that the preparation of the KEYS file can be found in [release preparation](. /release-prepare.md).

4. Polling

Initiate a poll in the dev@doris mailgroup, with the following template.

  1. Hi All,
  3. This is a call for vote to release Flink Connectors v1.0.0 for Apache Doris(Incubating).
  4. There are 3 outputs for different Flink version:
  6. - apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating
  7. - apache-doris-flink-connector-1.12.7-2.12-1.0.0-incubating
  8. - apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating
  10. The release node:
  11. xxxxx
  13. The release candidates:
  14. https://dist.apache.org/repos/dist/dev/incubator/doris/flink-connector/1.0.0/
  16. Maven 2 staging repository:
  17. https://repository.apache.org/content/repositories/orgapachedoris-1002/org/apache/doris/doris-flink-connector/
  19. Git tag for the release:
  20. https://github.com/apache/incubator-doris-flink-connector/tree/1.11.6-2.12-1.0.0
  21. https://github.com/apache/incubator-doris-flink-connector/tree/1.12.7-2.12-1.0.0
  22. https://github.com/apache/incubator-doris-flink-connector/tree/1.13.5-2.12-1.0.0
  24. Keys to verify the Release Candidate:
  25. https://downloads.apache.org/incubator/doris/KEYS
  27. Look at here for how to verify this release candidate:
  28. http://doris.incubator.apache.org/community/release-and-verify/release-verify.html
  30. The vote will be open for at least 72 hours or until necessary number of votes are reached.
  32. Please vote accordingly:
  34. [ ] +1 approve
  35. [ ] +0 no opinion
  36. [ ] -1 disapprove with the reason

After the dev mail group is approved, send an email to the general@incubator mail group for IPMC voting.

  1. Hi All,
  3. This is a call for vote to release Flink Connectors v1.0.0 for Apache Doris(Incubating).
  4. There are 3 outputs for different Flink version:
  6. - apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating
  7. - apache-doris-flink-connector-1.12.7-2.12-1.0.0-incubating
  8. - apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating
  10. The release node:
  11. xxxxx
  13. The release candidates:
  14. https://dist.apache.org/repos/dist/dev/incubator/doris/flink-connector/1.0.0/
  16. Maven 2 staging repository:
  17. https://repository.apache.org/content/repositories/orgapachedoris-1002/org/apache/doris/doris-flink-connector/
  19. Git tag for the release:
  20. https://github.com/apache/incubator-doris-flink-connector/tree/1.11.6-2.12-1.0.0
  21. https://github.com/apache/incubator-doris-flink-connector/tree/1.12.7-2.12-1.0.0
  22. https://github.com/apache/incubator-doris-flink-connector/tree/1.13.5-2.12-1.0.0
  24. Keys to verify the Release Candidate:
  25. https://downloads.apache.org/incubator/doris/KEYS
  27. Look at here for how to verify this release candidate:
  28. http://doris.incubator.apache.org/community/release-and-verify/release-verify.html
  30. Vote thread at dev@doris: [1]
  32. The vote will be open for at least 72 hours or until necessary number of votes are reached.
  34. Please vote accordingly:
  36. [ ] +1 approve
  37. [ ] +0 no opinion
  38. [ ] -1 disapprove with the reason
  40. [1] vote thread in dev@doris

Completing the release

Please refer to the Release Completion document to complete the release process.

Appendix: Releasing to SNAPSHOT

Snapshot is not an Apache Release version and is only used for pre-release previews. Snapshot versions can be released after discussion and approval by the PMC

Switch to the flink connector directory, we will use flink version 1.13.5, scalar 2.12 as an example

  1. cd flink-doris-connector
  2. mvn deploy -Dflink.version=1.13.5 -Dscala.version=2.12

After that you can see the snapshot version here.

  1. https://repository.apache.org/content/repositories/snapshots/org/apache/doris/doris-flink-connector/