Seatunnel

The newest Apache SeaTunnel (formerly waterdrop ) already supports Doris’s connector, SeaTunnel can use Spark engine and Flink engine to synchronize data to Doris.

Seatunnel Flink Sink Doris plugin code

Options

nametyperequireddefault valueengine
fenodesstringyes-Flink
databasestringyes-Flink
tablestringyes-Flink
userstringyes-Flink
passwordstringyes-Flink
batch_sizeintno100Flink
intervalintno1000Flink
max_retriesintno1Flink
doris.*-no-Flink

fenodes [string]

Doris Fe http url, eg: 127.0.0.1:8030

database [string]

Doris database

table [string]

Doris table

user [string]

Doris user

password [string]

Doris password

batch_size [int]

The maximum number of lines to write to Doris at a time, the default value is 100

interval [int]

The flush interval (in milliseconds), after which the asynchronous thread writes the data in the cache to Doris. Set to 0 to turn off periodic writes.

max_retries [int]

Number of retries after writing to Doris fails

doris.* [string]

Import parameters for Stream load. For example: ‘doris.column_separator’ = ‘, ‘ etc.

More Stream Load parameter configuration

Examples

Socket To Doris

  1. env {
  2. execution.parallelism = 1
  3. }
  4. source {
  5. SocketStream {
  6. host = 127.0.0.1
  7. port = 9999
  8. result_table_name = "socket"
  9. field_name = "info"
  10. }
  11. }
  12. transform {
  13. }
  14. sink {
  15. DorisSink {
  16. fenodes = "127.0.0.1:8030"
  17. user = root
  18. password = 123456
  19. database = test
  20. table = test_tbl
  21. batch_size = 5
  22. max_retries = 1
  23. interval = 5000
  24. }
  25. }

Start command

  1. sh bin/start-seatunnel-flink.sh --config config/flink.streaming.conf