Flink Doris Connector

This document applies to flink-doris-connector versions after 1.1.0, for versions before 1.1.0 refer to here

The Flink Doris Connector can support operations (read, insert, modify, delete) data stored in Doris through Flink.

Github: https://github.com/apache/doris-flink-connector

  • Doris table can be mapped to DataStream or Table.

Note:

  1. Modification and deletion are only supported on the Unique Key model
  2. The current deletion is to support Flink CDC to access data to achieve automatic deletion. If it is to delete other data access methods, you need to implement it yourself. For the data deletion usage of Flink CDC, please refer to the last section of this document

Version Compatibility

Connector VersionFlink VersionDoris VersionJava VersionScala Version
1.0.31.11+0.15+82.11,2.12
1.1.01.141.0+82.11,2.12
1.2.01.151.0+8-
1.3.01.161.0+8-

Build and Install

Ready to work

1.Modify the custom_env.sh.tpl file and rename it to custom_env.sh

2.Specify the thrift installation directory

  1. ##source file content
  2. #export THRIFT_BIN=
  3. #export MVN_BIN=
  4. #export JAVA_HOME=
  5. ##amend as below,MacOS as an example
  6. export THRIFT_BIN=/opt/homebrew/Cellar/thrift@0.13.0/0.13.0/bin/thrift
  7. #export MVN_BIN=
  8. #export JAVA_HOME=

Install thrift 0.13.0 (Note: Doris 0.15 and the latest builds are based on thrift 0.13.0, previous versions are still built with thrift 0.9.3) Windows:

  1. Download: http://archive.apache.org/dist/thrift/0.13.0/thrift-0.13.0.exe
  2. Modify thrift-0.13.0.exe to thrift

MacOS:

  1. Download: brew install thrift@0.13.0
  2. default address: /opt/homebrew/Cellar/thrift@0.13.0/0.13.0/bin/thrift

Note: Executing brew install thrift@0.13.0 on MacOS may report an error that the version cannot be found. The solution is as follows, execute it in the terminal:

  1. 1. `brew tap-new $USER/local-tap`
  2. 2. `brew extract --version='0.13.0' thrift $USER/local-tap`
  3. 3. `brew install thrift@0.13.0`

Reference link: https://gist.github.com/tonydeng/02e571f273d6cce4230dc8d5f394493c

Linux: 1.Download source package:wget https://archive.apache.org/dist/thrift/0.13.0/thrift-0.13.0.tar.gz 2.Install dependencies:yum install -y autoconf automake libtool cmake ncurses-devel openssl-devel lzo-devel zlib-devel gcc gcc-c++ 3.tar zxvf thrift-0.13.0.tar.gz 4.cd thrift-0.13.0 5../configure --without-tests 6.make 7.make install Check the version after installation is complete:thrift --version Note: If you have compiled Doris, you do not need to install thrift, you can directly use $DORIS_HOME/thirdparty/installed/bin/thrift

Execute following command in source dir:

  1. sh build.sh
  2. Usage:
  3. build.sh --flink version # specify flink version (after flink-doris-connector v1.2 and flink-1.15, there is no need to provide scala version)
  4. build.sh --tag # this is a build from tag
  5. e.g.:
  6. build.sh --flink 1.16.0
  7. build.sh --tag

Then, for example, execute the command to compile according to the version you need: sh build.sh --flink 1.16.0

After successful compilation, the file flink-doris-connector-1.16-1.3.0-SNAPSHOT.jar will be generated in the target/ directory. Copy this file to classpath in Flink to use Flink-Doris-Connector. For example, Flink running in Local mode, put this file in the lib/ folder. Flink running in Yarn cluster mode, put this file in the pre-deployment package.

Remarks:

  1. Doris FE should be configured to enable http v2 in the configuration

    conf/fe.conf

  1. enable_http_server_v2 = true

Using Maven

Add flink-doris-connector Maven dependencies

  1. <!-- flink-doris-connector -->
  2. <dependency>
  3. <groupId>org.apache.doris</groupId>
  4. <artifactId>flink-doris-connector-1.16</artifactId>
  5. <version>1.3.0</version>
  6. </dependency>

Notes

  1. Please replace the corresponding Connector and Flink dependency versions according to different Flink versions. Version 1.3.0 only supports Flink1.16

  2. You can also download the relevant version jar package from here.

How to use

There are three ways to use Flink Doris Connector.

  • SQL
  • DataStream

Parameters Configuration

Flink Doris Connector Sink writes data to Doris by the Stream load, and also supports the configurations of Stream load, For specific parameters, please refer to here.

  • SQL configured by sink.properties. in the WITH
  • DataStream configured by DorisExecutionOptions.builder().setStreamLoadProp(Properties)

SQL

  • Source
  1. CREATE TABLE flink_doris_source (
  2. name STRING,
  3. age INT,
  4. price DECIMAL(5,2),
  5. sale DOUBLE
  6. )
  7. WITH (
  8. 'connector' = 'doris',
  9. 'fenodes' = 'FE_IP:8030',
  10. 'table.identifier' = 'database.table',
  11. 'username' = 'root',
  12. 'password' = 'password'
  13. );
  • Sink
  1. -- enable checkpoint
  2. SET 'execution.checkpointing.interval' = '10s';
  3. CREATE TABLE flink_doris_sink (
  4. name STRING,
  5. age INT,
  6. price DECIMAL(5,2),
  7. sale DOUBLE
  8. )
  9. WITH (
  10. 'connector' = 'doris',
  11. 'fenodes' = 'FE_IP:8030',
  12. 'table.identifier' = 'db.table',
  13. 'username' = 'root',
  14. 'password' = 'password',
  15. 'sink.label-prefix' = 'doris_label'
  16. );
  • Insert
  1. INSERT INTO flink_doris_sink select name,age,price,sale from flink_doris_source

DataStream

  • Source
  1. DorisOptions.Builder builder = DorisOptions.builder()
  2. .setFenodes("FE_IP:8030")
  3. .setTableIdentifier("db.table")
  4. .setUsername("root")
  5. .setPassword("password");
  6. DorisSource<List<?>> dorisSource = DorisSourceBuilder.<List<?>>builder()
  7. .setDorisOptions(builder.build())
  8. .setDorisReadOptions(DorisReadOptions.builder().build())
  9. .setDeserializer(new SimpleListDeserializationSchema())
  10. .build();
  11. env.fromSource(dorisSource, WatermarkStrategy.noWatermarks(), "doris source").print();
  • Sink

String Stream

  1. // enable checkpoint
  2. env.enableCheckpointing(10000);
  3. // using batch mode for bounded data
  4. env.setRuntimeMode(RuntimeExecutionMode.BATCH);
  5. DorisSink.Builder<String> builder = DorisSink.builder();
  6. DorisOptions.Builder dorisBuilder = DorisOptions.builder();
  7. dorisBuilder.setFenodes("FE_IP:8030")
  8. .setTableIdentifier("db.table")
  9. .setUsername("root")
  10. .setPassword("password");
  11. Properties properties = new Properties();
  12. /**
  13. json format to streamload
  14. properties.setProperty("format", "json");
  15. properties.setProperty("read_json_by_line", "true");
  16. **/
  17. DorisExecutionOptions.Builder executionBuilder = DorisExecutionOptions.builder();
  18. executionBuilder.setLabelPrefix("label-doris") //streamload label prefix
  19. .setStreamLoadProp(properties);
  20. builder.setDorisReadOptions(DorisReadOptions.builder().build())
  21. .setDorisExecutionOptions(executionBuilder.build())
  22. .setSerializer(new SimpleStringSerializer()) //serialize according to string
  23. .setDorisOptions(dorisBuilder.build());
  24. //mock string source
  25. List<Tuple2<String, Integer>> data = new ArrayList<>();
  26. data.add(new Tuple2<>("doris",1));
  27. DataStreamSource<Tuple2<String, Integer>> source = env.fromCollection(data);
  28. source.map((MapFunction<Tuple2<String, Integer>, String>) t -> t.f0 + "\t" + t.f1)
  29. .sinkTo(builder.build());

RowData Stream

  1. // enable checkpoint
  2. env.enableCheckpointing(10000);
  3. // using batch mode for bounded data
  4. env.setRuntimeMode(RuntimeExecutionMode.BATCH);
  5. //doris sink option
  6. DorisSink.Builder<RowData> builder = DorisSink.builder();
  7. DorisOptions.Builder dorisBuilder = DorisOptions.builder();
  8. dorisBuilder.setFenodes("FE_IP:8030")
  9. .setTableIdentifier("db.table")
  10. .setUsername("root")
  11. .setPassword("password");
  12. // json format to streamload
  13. Properties properties = new Properties();
  14. properties.setProperty("format", "json");
  15. properties.setProperty("read_json_by_line", "true");
  16. DorisExecutionOptions.Builder executionBuilder = DorisExecutionOptions.builder();
  17. executionBuilder.setLabelPrefix("label-doris") //streamload label prefix
  18. .setStreamLoadProp(properties); //streamload params
  19. //flink rowdata‘s schema
  20. String[] fields = {"city", "longitude", "latitude", "destroy_date"};
  21. DataType[] types = {DataTypes.VARCHAR(256), DataTypes.DOUBLE(), DataTypes.DOUBLE(), DataTypes.DATE()};
  22. builder.setDorisReadOptions(DorisReadOptions.builder().build())
  23. .setDorisExecutionOptions(executionBuilder.build())
  24. .setSerializer(RowDataSerializer.builder() //serialize according to rowdata
  25. .setFieldNames(fields)
  26. .setType("json") //json format
  27. .setFieldType(types).build())
  28. .setDorisOptions(dorisBuilder.build());
  29. //mock rowdata source
  30. DataStream<RowData> source = env.fromElements("")
  31. .map(new MapFunction<String, RowData>() {
  32. @Override
  33. public RowData map(String value) throws Exception {
  34. GenericRowData genericRowData = new GenericRowData(4);
  35. genericRowData.setField(0, StringData.fromString("beijing"));
  36. genericRowData.setField(1, 116.405419);
  37. genericRowData.setField(2, 39.916927);
  38. genericRowData.setField(3, LocalDate.now().toEpochDay());
  39. return genericRowData;
  40. }
  41. });
  42. source.sinkTo(builder.build());

General

KeyDefault ValueRequiredComment
fenodesYDoris FE http address, support multiple addresses, separated by commas
table.identifierYDoris table identifier, eg, db1.tbl1
usernameYDoris username
passwordYDoris password
doris.request.retries3NNumber of retries to send requests to Doris
doris.request.connect.timeout.ms30000NConnection timeout for sending requests to Doris
doris.request.read.timeout.ms30000NRead timeout for sending request to Doris
doris.request.query.timeout.s3600NQuery the timeout time of doris, the default is 1 hour, -1 means no timeout limit
doris.request.tablet.sizeInteger.MAX_VALUENThe number of Doris Tablets corresponding to an Partition. The smaller this value is set, the more partitions will be generated. This will increase the parallelism on the flink side, but at the same time will cause greater pressure on Doris.
doris.batch.size1024NThe maximum number of rows to read data from BE at one time. Increasing this value can reduce the number of connections between Flink and Doris. Thereby reducing the extra time overhead caused by network delay.
doris.exec.mem.limit2147483648NMemory limit for a single query. The default is 2GB, in bytes.
doris.deserialize.arrow.asyncfalseNWhether to support asynchronous conversion of Arrow format to RowBatch required for flink-doris-connector iteration
doris.deserialize.queue.size64NAsynchronous conversion of the internal processing queue in Arrow format takes effect when doris.deserialize.arrow.async is true
doris.read.fieldNList of column names in the Doris table, separated by commas
doris.filter.queryNFilter expression of the query, which is transparently transmitted to Doris. Doris uses this expression to complete source-side data filtering.
sink.label-prefixYThe label prefix used by stream load imports. In the 2pc scenario, global uniqueness is required to ensure the EOS semantics of Flink.
sink.properties.*NThe stream load parameters.

eg:
sink.properties.column_separator’ = ‘,’

Setting ‘sink.properties.escape_delimiters’ = ‘true’ if you want to use a control char as a separator, so that such as ‘\x01’ will translate to binary 0x01

Support JSON format import, you need to enable both ‘sink.properties.format’ =’json’ and ‘sink.properties.strip_outer_array’ =’true’
sink.enable-deletetrueNWhether to enable deletion. This option requires Doris table to enable batch delete function (0.15+ version is enabled by default), and only supports Uniq model.
sink.enable-2pctrueNWhether to enable two-phase commit (2pc), the default is true, to ensure Exactly-Once semantics. For two-phase commit, please refer to here.
sink.max-retries1NIn the 2pc scenario, the number of retries after the commit phase fails.
sink.buffer-size1048576(1MB)NWrite data cache buffer size, in bytes. It is not recommended to modify, the default configuration is sufficient.
sink.buffer-count3NThe number of write data cache buffers, it is not recommended to modify, the default configuration is sufficient.
Doris TypeFlink Type
NULL_TYPENULL
BOOLEANBOOLEAN
TINYINTTINYINT
SMALLINTSMALLINT
INTINT
BIGINTBIGINT
FLOATFLOAT
DOUBLEDOUBLE
DATEDATE
DATETIMETIMESTAMP
DECIMALDECIMAL
CHARSTRING
LARGEINTSTRING
VARCHARSTRING
DECIMALV2DECIMAL
TIMEDOUBLE
HLLUnsupported datatype
  1. SET 'execution.checkpointing.interval' = '10s';
  2. CREATE TABLE cdc_mysql_source (
  3. id int
  4. ,name VARCHAR
  5. ,PRIMARY KEY (id) NOT ENFORCED
  6. ) WITH (
  7. 'connector' = 'mysql-cdc',
  8. 'hostname' = '127.0.0.1',
  9. 'port' = '3306',
  10. 'username' = 'root',
  11. 'password' = 'password',
  12. 'database-name' = 'database',
  13. 'table-name' = 'table'
  14. );
  15. -- Support delete event synchronization (sink.enable-delete='true'), requires Doris table to enable batch delete function
  16. CREATE TABLE doris_sink (
  17. id INT,
  18. name STRING
  19. )
  20. WITH (
  21. 'connector' = 'doris',
  22. 'fenodes' = '127.0.0.1:8030',
  23. 'table.identifier' = 'database.table',
  24. 'username' = 'root',
  25. 'password' = '',
  26. 'sink.properties.format' = 'json',
  27. 'sink.properties.read_json_by_line' = 'true',
  28. 'sink.enable-delete' = 'true',
  29. 'sink.label-prefix' = 'doris_label'
  30. );
  31. insert into doris_sink select id,name from cdc_mysql_source;

Java example

samples/doris-demo/ An example of the Java version is provided below for reference, see here

Best Practices

Application scenarios

The most suitable scenario for using Flink Doris Connector is to synchronize source data to Doris (Mysql, Oracle, PostgreSQL) in real time/batch, etc., and use Flink to perform joint analysis on data in Doris and other data sources. You can also use Flink Doris Connector

Other

  1. The Flink Doris Connector mainly relies on Checkpoint for streaming writing, so the interval between Checkpoints is the visible delay time of the data.
  2. To ensure the Exactly Once semantics of Flink, the Flink Doris Connector enables two-phase commit by default, and Doris enables two-phase commit by default after version 1.1. 1.0 can be enabled by modifying the BE parameters, please refer to two_phase_commit.

FAQ

  1. After Doris Source finishes reading data, why does the stream end?

Currently Doris Source is a bounded stream and does not support CDC reading.

  1. Can Flink read Doris and perform conditional pushdown?

By configuring the doris.filter.query parameter, refer to the configuration section for details.

  1. How to write Bitmap type?
  1. CREATE TABLE bitmap_sink (
  2. dt int,
  3. page string,
  4. user_id int
  5. )
  6. WITH (
  7. 'connector' = 'doris',
  8. 'fenodes' = '127.0.0.1:8030',
  9. 'table.identifier' = 'test.bitmap_test',
  10. 'username' = 'root',
  11. 'password' = '',
  12. 'sink.label-prefix' = 'doris_label',
  13. 'sink.properties.columns' = 'dt,page,user_id,user_id=to_bitmap(user_id)'
  14. )
  1. errCode = 2, detailMessage = Label [label_0_1] has already been used, relate to txn [19650]

In the Exactly-Once scenario, the Flink Job must be restarted from the latest Checkpoint/Savepoint, otherwise the above error will be reported. When Exactly-Once is not required, it can also be solved by turning off 2PC commits (sink.enable-2pc=false) or changing to a different sink.label-prefix.

  1. errCode = 2, detailMessage = transaction [19650] not found

Occurred in the Commit phase, the transaction ID recorded in the checkpoint has expired on the FE side, and the above error will occur when committing again at this time. At this time, it cannot be started from the checkpoint, and the expiration time can be extended by modifying the streaming_label_keep_max_second configuration in fe.conf, which defaults to 12 hours.

  1. errCode = 2, detailMessage = current running txns on db 10006 is 100, larger than limit 100

This is because the concurrent import of the same library exceeds 100, which can be solved by adjusting the parameter max_running_txn_num_per_db of fe.conf. For details, please refer to max_running_txn_num_per_db

  1. How to ensure the order of a batch of data when Flink writes to the Uniq model?

You can add sequence column configuration to ensure that, for details, please refer to sequence

  1. The Flink task does not report an error, but the data cannot be synchronized?

Before Connector1.1.0, it was written in batches, and the writing was driven by data. It was necessary to determine whether there was data written upstream. After 1.1.0, it depends on Checkpoint, and Checkpoint must be enabled to write.

  1. tablet writer write failed, tablet_id=190958, txn_id=3505530, err=-235

It usually occurs before Connector1.1.0, because the writing frequency is too fast, resulting in too many versions. The frequency of Streamload can be reduced by setting the sink.batch.size and sink.batch.interval parameters.