Seatunnel

The newest Apache SeaTunnel (waterdop) (opens new window) has supported Doris connector, seatunnel can load data by Spark engine or Flink engine.

In fact,seatunnel load data by stream load function.Everyone is welcome to use

Install Seatunnel

Seatunnel install (opens new window)

Spark Sink Doris

Options

fenodes [string]

Doris FE address:8030

database [string]

Doris target database name

table [string]

Doris target table name

user [string]

Doris user name

password [string]

Doris user’s password

batch_size [string]

Doris number of submissions per batch

doris. [string] Doris stream_load properties,you can use ‘doris.’ prefix + stream_load properties

More Doris stream_load Configurations (opens new window)

Examples

Hive to Doris

Config properties

env{
  spark.app.name = "hive2doris-template"
}
spark {
  spark.sql.catalogImplementation = "hive"
}
source {
  hive {
    preSql = "select * from tmp.test"
    result_table_name = "test"
  }
}
transform {
}
sink {
Console {
  }
Doris {
   fenodes="xxxx:8030"
   database="gl_mint_dim"
   table="dim_date"
   user="root"
   password="root"
   batch_size=1000
   doris.column_separator="\t"
   doris.columns="date_key,date_value,day_in_year,day_in_month"
   }
}

Start command

sh bin/start-waterdrop-spark.sh --master local[4] --deploy-mode client --config ./config/spark.conf