Datasource Setting

Standalone Switching Metadata Database Configuration

We here use MySQL as an example to illustrate how to configure an external database:

NOTE: If you use MySQL, you need to manually download mysql-connector-java driver (8.0.16) and move it to the libs directory of DolphinScheduler which is api-server/libs and alert-server/libs and master-server/libs and worker-server/libs.

  • First of all, follow the instructions in datasource-setting Pseudo-Cluster/Cluster Initialize the Database section to create and initialize database
  • Set the following environment variables in your terminal or modify the bin/env/dolphinscheduler_env.sh with your database username and password for {user} and {password}:
  1. export DATABASE=mysql
  2. export SPRING_PROFILES_ACTIVE=${DATABASE}
  3. export SPRING_DATASOURCE_USERNAME={user}
  4. export SPRING_DATASOURCE_PASSWORD={password}
  • Add mysql-connector-java driver to ./standalone-server/libs/standalone-server/, see general-setting Pseudo-Cluster/Cluster Initialize the Database section about where to download
  • Start standalone-server, now you are using mysql as database and it will not clear up your data when you stop or restart standalone-server.

Pseudo-Cluster/Cluster Initialize the Database

DolphinScheduler stores metadata in relational database. Currently, we support PostgreSQL and MySQL. Let’s walk through how to initialize the database in MySQL and PostgreSQL :

If you use MySQL, you need to manually download mysql-connector-java driver (8.0.16) and move it to the libs directory of DolphinScheduler which is api-server/libs and alert-server/libs and master-server/libs and worker-server/libs.

For mysql 5.6 / 5.7

  1. mysql -uroot -p
  3. mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
  5. # Replace {user} and {password} with your username and password
  6. mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}';
  7. mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY '{password}';
  9. mysql> flush privileges;

For mysql 8:

  1. mysql -uroot -p
  3. mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
  5. # Replace {user} and {password} with your username and password
  6. mysql> CREATE USER '{user}'@'%' IDENTIFIED BY '{password}';
  7. mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%';
  8. mysql> CREATE USER '{user}'@'localhost' IDENTIFIED BY '{password}';
  9. mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost';
  10. mysql> FLUSH PRIVILEGES;

For PostgreSQL:

  1. # Use psql-tools to login PostgreSQL
  2. psql
  3. # Create a database
  4. postgres=# CREATE DATABASE dolphinscheduler;
  5. # Replace {user} and {password} with your username and password
  6. postgres=# CREATE USER {user} PASSWORD {password};
  7. postgres=# ALTER DATABASE dolphinscheduler OWNER TO {user};
  8. # Logout PostgreSQL
  9. postgres=#\q
  10. # Exec cmd below in terminal, add config to pg_hba.conf and reload PostgreSQL config, replace {ip} to DS cluster ip addresses
  11. echo "host    dolphinscheduler   {user}    {ip}     md5" >> $PGDATA/pg_hba.conf
  12. pg_ctl reload

Then, modify ./bin/env/dolphinscheduler_env.sh, change {user} and {password} to what you set in the previous step.

For MySQL:

  1. # for mysql
  2. export DATABASE=${DATABASE:-mysql}
  3. export SPRING_PROFILES_ACTIVE=${DATABASE}
  4. export SPRING_DATASOURCE_URL="jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false"
  5. export SPRING_DATASOURCE_USERNAME={user}
  6. export SPRING_DATASOURCE_PASSWORD={password}

For PostgreSQL:

  1. # for postgresql
  2. export DATABASE=${DATABASE:-postgresql}
  3. export SPRING_PROFILES_ACTIVE=${DATABASE}
  4. export SPRING_DATASOURCE_URL="jdbc:postgresql://127.0.0.1:5432/dolphinscheduler"
  5. export SPRING_DATASOURCE_USERNAME={user}
  6. export SPRING_DATASOURCE_PASSWORD={password}

After the above steps done you would create a new database for DolphinScheduler, then run the Shell script to init database:

  1. bash tools/bin/upgrade-schema.sh

DataSource Center

DataSource supports MySQL, PostgreSQL, Hive/Impala, Spark, ClickHouse, Oracle, SQL Server and other DataSources.

  • Click bottom Data Source Center -> Create Data Source to create a new datasource.
  • Click Test Connection to test whether the DataSource can connect successfully (datasource can be saved only if it passes the connection test).

Using datasource incompatible to Apache LICENSE V2 LICENSE

Some of datasource are native supported to DolphinScheduler while others need users download JDBC driver package manually, because those JDBC driver incompatible to Apache LICENSE V2 LICENSE. For this reason we have to release DolphinScheduler’s distribute package without those packages, even if this will make more complicated for users. Datasource such as MySQL, Oracle, SQL Server as the examples, but we have the solution to solve this

Example

For example, if you want to use MySQL datasource, you need to download the correct JDBC driver from mysql maven repository, and move it into directory api-server/libs and worker-server/libs. After that, you could activate MySQL datasource by restarting api-server and worker-server. Mount to container volume in the same path and restart it if you use container like Docker.

Note: If you only want to use MySQL in the datasource center, there is no requirement for the version of MySQL JDBC driver. But if you want to use MySQL as the metabase of DolphinScheduler, it only supports [8.0.16 and above](https:/ /repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar) version.