Seatunnel

The newest Apache SeaTunnel (waterdop) has supported Doris connector, seatunnel can load data by Spark engine or Flink engine.

In fact,seatunnel load data by stream load function.Everyone is welcome to use

Install Seatunnel

Seatunnel install

Spark Sink Doris

Options

nametyperequireddefault valueengine
fenodesstringyes-Spark
databasestringyes-Spark
tablestringyes-Spark
userstringyes-Spark
passwordstringyes-Spark
batch_sizeintyes100Spark
doris.*stringno-Spark

fenodes [string]

Doris FE address:8030

database [string]

Doris target database name

table [string]

Doris target table name

user [string]

Doris user name

password [string]

Doris user’s password

batch_size [string]

Doris number of submissions per batch

doris. [string] Doris stream_load properties,you can use ‘doris.’ prefix + stream_load properties

More Doris stream_load Configurations

Examples

Hive to Doris

Config properties

  1. env{
  2. spark.app.name = "hive2doris-template"
  3. }
  4. spark {
  5. spark.sql.catalogImplementation = "hive"
  6. }
  7. source {
  8. hive {
  9. preSql = "select * from tmp.test"
  10. result_table_name = "test"
  11. }
  12. }
  13. transform {
  14. }
  15. sink {
  16. Console {
  17. }
  18. Doris {
  19. fenodes="xxxx:8030"
  20. database="gl_mint_dim"
  21. table="dim_date"
  22. user="root"
  23. password="root"
  24. batch_size=1000
  25. doris.column_separator="\t"
  26. doris.columns="date_key,date_value,day_in_year,day_in_month"
  27. }
  28. }

Start command

  1. sh bin/start-waterdrop-spark.sh --master local[4] --deploy-mode client --config ./config/spark.conf