Provisioning Grafana

In previous versions of Grafana, you could only use the API for provisioning data sources and dashboards. But that required the service to be running before you started creating dashboards and you also needed to set up credentials for the HTTP API. In v5.0 we decided to improve this experience by adding a new active provisioning system that uses config files. This will make GitOps more natural as data sources and dashboards can be defined via files that can be version controlled. We hope to extend this system to later add support for users, orgs and alerts as well.

Config File

Check out the configuration page for more information on what you can configure in grafana.ini

Config File Locations

  • Default configuration from $WORKING_DIR/conf/defaults.ini
  • Custom configuration from $WORKING_DIR/conf/custom.ini
  • The custom configuration file path can be overridden using the --config parameter

Note: If you have installed Grafana using the deb or rpm packages, then your configuration file is located at /etc/grafana/grafana.ini. This path is specified in the Grafana init.d script using --config file parameter.

Using Environment Variables

It is possible to use environment variable interpolation in all 3 provisioning configuration types. Allowed syntax is either $ENV_VAR_NAME or ${ENV_VAR_NAME} and can be used only for values not for keys or bigger parts of the configurations. It is not available in the dashboard’s definition files just the dashboard provisioning configuration. Example:

  1. datasources:
  2. - name: Graphite
  3. url: http://localhost:$PORT
  4. user: $USER
  5. secureJsonData:
  6. password: $PASSWORD

If you have a literal $ in your value and want to avoid interpolation, $$ can be used.


Configuration Management Tools

Currently we do not provide any scripts/manifests for configuring Grafana. Rather than spending time learning and creating scripts/manifests for each tool, we think our time is better spent making Grafana easier to provision. Therefore, we heavily rely on the expertise of the community.

ToolProject
Puppethttps://forge.puppet.com/puppet/grafana
Ansiblehttps://github.com/cloudalchemy/ansible-grafana
Chefhttps://github.com/JonathanTron/chef-grafana
Saltstackhttps://github.com/salt-formulas/salt-formula-grafana
Jsonnethttps://github.com/grafana/grafonnet-lib/

Data sources

This feature is available from v5.0

It’s possible to manage data sources in Grafana by adding one or more YAML config files in the provisioning/datasources directory. Each config file can contain a list of datasources that will get added or updated during start up. If the data source already exists, then Grafana updates it to match the configuration file. The config file can also contain a list of data sources that should be deleted. That list is called deleteDatasources. Grafana will delete data sources listed in deleteDatasources before inserting/updating those in the datasource list.

Running Multiple Grafana Instances

If you are running multiple instances of Grafana you might run into problems if they have different versions of the datasource.yaml configuration file. The best way to solve this problem is to add a version number to each datasource in the configuration and increase it when you update the config. Grafana will only update datasources with the same or lower version number than specified in the config. That way, old configs cannot overwrite newer configs if they restart at the same time.

Example data source Config File

  1. # config file version
  2. apiVersion: 1
  3. # list of datasources that should be deleted from the database
  4. deleteDatasources:
  5. - name: Graphite
  6. orgId: 1
  7. # list of datasources to insert/update depending
  8. # what's available in the database
  9. datasources:
  10. # <string, required> name of the datasource. Required
  11. - name: Graphite
  12. # <string, required> datasource type. Required
  13. type: graphite
  14. # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
  15. access: proxy
  16. # <int> org id. will default to orgId 1 if not specified
  17. orgId: 1
  18. # <string> custom UID which can be used to reference this datasource in other parts of the configuration, if not specified will be generated automatically
  19. uid: my_unique_uid
  20. # <string> url
  21. url: http://localhost:8080
  22. # <string> Deprecated, use secureJsonData.password
  23. password:
  24. # <string> database user, if used
  25. user:
  26. # <string> database name, if used
  27. database:
  28. # <bool> enable/disable basic auth
  29. basicAuth:
  30. # <string> basic auth username
  31. basicAuthUser:
  32. # <string> Deprecated, use secureJsonData.basicAuthPassword
  33. basicAuthPassword:
  34. # <bool> enable/disable with credentials headers
  35. withCredentials:
  36. # <bool> mark as default datasource. Max one per org
  37. isDefault:
  38. # <map> fields that will be converted to json and stored in jsonData
  39. jsonData:
  40. graphiteVersion: '1.1'
  41. tlsAuth: true
  42. tlsAuthWithCACert: true
  43. # <string> json object of data that will be encrypted.
  44. secureJsonData:
  45. tlsCACert: '...'
  46. tlsClientCert: '...'
  47. tlsClientKey: '...'
  48. # <string> database password, if used
  49. password:
  50. # <string> basic auth password
  51. basicAuthPassword:
  52. version: 1
  53. # <bool> allow users to edit datasources from the UI.
  54. editable: false

Custom Settings per Datasource

Please refer to each datasource documentation for specific provisioning examples.

DatasourceMisc
ElasticsearchElasticsearch uses the database property to configure the index for a datasource

JSON Data

Since not all datasources have the same configuration settings we only have the most common ones as fields. The rest should be stored as a json blob in the jsonData field. Here are the most common settings that the core datasources use.

Note: Datasources tagged with HTTP* below denotes any data source which communicates using the HTTP protocol, e.g. all core data source plugins except MySQL, PostgreSQL and MSSQL.

NameTypeDatasourceDescription
tlsAuthbooleanHTTP, MySQLEnable TLS authentication using client cert configured in secure json data
tlsAuthWithCACertbooleanHTTP, MySQL, PostgreSQLEnable TLS authentication using CA cert
tlsSkipVerifybooleanHTTP, MySQL, PostgreSQLControls whether a client verifies the server’s certificate chain and host name.
serverNamestringHTTPOptional. Controls the server name used for certificate common name/subject alternative name verification. Defaults to using the data source URL.
timeoutstringHTTPRequest timeout in seconds. Overrides dataproxy.timeout option
graphiteVersionstringGraphiteGraphite version
timeIntervalstringPrometheus, Elasticsearch, InfluxDB, MySQL, PostgreSQL and MSSQLLowest interval/step value that should be used for this data source.
httpModestringInfluxdbHTTP Method. ‘GET’, ‘POST’, defaults to GET
maxSeriesnumberInfluxdbMax number of series/tables that Grafana processes
httpMethodstringPrometheusHTTP Method. ‘GET’, ‘POST’, defaults to POST
customQueryParametersstringPrometheusQuery parameters to add, as a URL-encoded string.
manageAlertsbooleanPrometheus and LokiManage alerts via Alerting UI
esVersionstringElasticsearchElasticsearch version (E.g. 7.0.0, 7.6.1)
timeFieldstringElasticsearchWhich field that should be used as timestamp
intervalstringElasticsearchIndex date time format. nil(No Pattern), ‘Hourly’, ‘Daily’, ‘Weekly’, ‘Monthly’ or ‘Yearly’
logMessageFieldstringElasticsearchWhich field should be used as the log message
logLevelFieldstringElasticsearchWhich field should be used to indicate the priority of the log message
sigV4AuthbooleanElasticsearch and PrometheusEnable usage of SigV4
sigV4AuthTypestringElasticsearch and PrometheusSigV4 auth provider. default/credentials/keys
sigV4ExternalIdstringElasticsearch and PrometheusOptional SigV4 External ID
sigV4AssumeRoleArnstringElasticsearch and PrometheusOptional SigV4 ARN role to assume
sigV4RegionstringElasticsearch and PrometheusSigV4 AWS region
sigV4ProfilestringElasticsearch and PrometheusOptional SigV4 credentials profile
authTypestringCloudwatchAuth provider. default/credentials/keys
externalIdstringCloudwatchOptional External ID
assumeRoleArnstringCloudwatchOptional ARN role to assume
defaultRegionstringCloudwatchOptional default AWS region
customMetricsNamespacesstringCloudwatchNamespaces of Custom Metrics
profilestringCloudwatchOptional credentials profile
tsdbVersionstringOpenTSDBVersion
tsdbResolutionstringOpenTSDBResolution
sslmodestringPostgreSQLSSLmode. ‘disable’, ‘require’, ‘verify-ca’ or ‘verify-full’
tlsConfigurationMethodstringPostgreSQLSSL Certificate configuration, either by ‘file-path’ or ‘file-content’
sslRootCertFilestringPostgreSQLSSL server root certificate file, must be readable by the Grafana user
sslCertFilestringPostgreSQLSSL client certificate file, must be readable by the Grafana user
sslKeyFilestringPostgreSQLSSL client key file, must be readable by only the Grafana user
encryptstringMSSQLConnection SSL encryption handling. ‘disable’, ‘false’ or ‘true’
postgresVersionnumberPostgreSQLPostgres version as a number (903/904/905/906/1000) meaning v9.3, v9.4, …, v10
timescaledbbooleanPostgreSQLEnable usage of TimescaleDB extension
maxOpenConnsnumberMySQL, PostgreSQL and MSSQLMaximum number of open connections to the database (Grafana v5.4+)
maxIdleConnsnumberMySQL, PostgreSQL and MSSQLMaximum number of connections in the idle connection pool (Grafana v5.4+)
connMaxLifetimenumberMySQL, PostgreSQL and MSSQLMaximum amount of time in seconds a connection may be reused (Grafana v5.4+)
keepCookiesarrayHTTPCookies that needs to be passed along while communicating with datasources

Secure Json Data

{"authType":"keys","defaultRegion":"us-west-2","timeField":"@timestamp"}

Secure json data is a map of settings that will be encrypted with secret key from the Grafana config. The purpose of this is only to hide content from the users of the application. This should be used for storing TLS Cert and password that Grafana will append to the request on the server side. All of these settings are optional.

Note: Datasources tagged with HTTP* below denotes any data source which communicates using the HTTP protocol, e.g. all core data source plugins except MySQL, PostgreSQL and MSSQL.

NameTypeDatasourceDescription
tlsCACertstringHTTP, MySQL, PostgreSQLCA cert for out going requests
tlsClientCertstringHTTP, MySQL, PostgreSQLTLS Client cert for outgoing requests
tlsClientKeystringHTTP, MySQL, PostgreSQLTLS Client key for outgoing requests
passwordstringHTTP, MySQL, PostgreSQL, MSSQLpassword
basicAuthPasswordstringHTTP*password for basic authentication
accessKeystringCloudwatchAccess key for connecting to Cloudwatch
secretKeystringCloudwatchSecret key for connecting to Cloudwatch
sigV4AccessKeystringElasticsearch and PrometheusSigV4 access key. Required when using keys auth provider
sigV4SecretKeystringElasticsearch and PrometheusSigV4 secret key. Required when using keys auth provider

Custom HTTP headers for datasources

Data sources managed by Grafanas provisioning can be configured to add HTTP headers to all requests going to that datasource. The header name is configured in the jsonData field and the header value should be configured in secureJsonData.

  1. apiVersion: 1
  2. datasources:
  3. - name: Graphite
  4. jsonData:
  5. httpHeaderName1: 'HeaderName'
  6. httpHeaderName2: 'Authorization'
  7. secureJsonData:
  8. httpHeaderValue1: 'HeaderValue'
  9. httpHeaderValue2: 'Bearer XXXXXXXXX'

Plugins

This feature is available from v7.1

You can manage plugins in Grafana by adding one or more YAML config files in the provisioning/plugins directory. Each config file can contain a list of apps that will be updated during start up. Grafana updates each app to match the configuration file.

Example plugin configuration file

  1. apiVersion: 1
  2. apps:
  3. # <string> the type of app, plugin identifier. Required
  4. - type: raintank-worldping-app
  5. # <int> Org ID. Default to 1, unless org_name is specified
  6. org_id: 1
  7. # <string> Org name. Overrides org_id unless org_id not specified
  8. org_name: Main Org.
  9. # <bool> disable the app. Default to false.
  10. disabled: false
  11. # <map> fields that will be converted to json and stored in jsonData. Custom per app.
  12. jsonData:
  13. # key/value pairs of string to object
  14. key: value
  15. # <map> fields that will be converted to json, encrypted and stored in secureJsonData. Custom per app.
  16. secureJsonData:
  17. # key/value pairs of string to string
  18. key: value

Dashboards

You can manage dashboards in Grafana by adding one or more YAML config files in the provisioning/dashboards directory. Each config file can contain a list of dashboards providers that load dashboards into Grafana from the local filesystem.

The dashboard provider config file looks somewhat like this:

  1. apiVersion: 1
  2. providers:
  3. # <string> an unique provider name. Required
  4. - name: 'a unique provider name'
  5. # <int> Org id. Default to 1
  6. orgId: 1
  7. # <string> name of the dashboard folder.
  8. folder: ''
  9. # <string> folder UID. will be automatically generated if not specified
  10. folderUid: ''
  11. # <string> provider type. Default to 'file'
  12. type: file
  13. # <bool> disable dashboard deletion
  14. disableDeletion: false
  15. # <int> how often Grafana will scan for changed dashboards
  16. updateIntervalSeconds: 10
  17. # <bool> allow updating provisioned dashboards from the UI
  18. allowUiUpdates: false
  19. options:
  20. # <string, required> path to dashboard files on disk. Required when using the 'file' type
  21. path: /var/lib/grafana/dashboards
  22. # <bool> use folder names from filesystem to create folders in Grafana
  23. foldersFromFilesStructure: true

When Grafana starts, it will update/insert all dashboards available in the configured path. Then later on poll that path every updateIntervalSeconds and look for updated json files and update/insert those into the database.

Note: Dashboards are provisioned to the General folder if the folder option is missing or empty.

Making changes to a provisioned dashboard

It’s possible to make changes to a provisioned dashboard in the Grafana UI. However, it is not possible to automatically save the changes back to the provisioning source. If allowUiUpdates is set to true and you make changes to a provisioned dashboard, you can Save the dashboard then changes will be persisted to the Grafana database.

Note: If a provisioned dashboard is saved from the UI and then later updated from the source, the dashboard stored in the database will always be overwritten. The version property in the JSON file will not affect this, even if it is lower than the existing dashboard.

If a provisioned dashboard is saved from the UI and the source is removed, the dashboard stored in the database will be deleted unless the configuration option disableDeletion is set to true.

If allowUiUpdates is configured to false, you are not able to make changes to a provisioned dashboard. When you click Save, Grafana brings up a Cannot save provisioned dashboard dialog. The screenshot below illustrates this behavior.

Grafana offers options to export the JSON definition of a dashboard. Either Copy JSON to Clipboard or Save JSON to file can help you synchronize your dashboard changes back to the provisioning source.

Note: The JSON definition in the input field when using Copy JSON to Clipboard or Save JSON to file will have the id field automatically removed to aid the provisioning workflow.

Provisioning - 图1Provisioning - 图2

Reusable Dashboard URLs

If the dashboard in the JSON file contains an UID, Grafana forces insert/update on that UID. This allows you to migrate dashboards between Grafana instances and provisioning Grafana from configuration without breaking the URLs given because the new dashboard URL uses the UID as identifier. When Grafana starts, it updates/inserts all dashboards available in the configured folders. If you modify the file, then the dashboard is also updated. By default, Grafana deletes dashboards in the database if the file is removed. You can disable this behavior using the disableDeletion setting.

Note: Provisioning allows you to overwrite existing dashboards which leads to problems if you re-use settings that are supposed to be unique. Be careful not to re-use the same title multiple times within a folder or uid within the same installation as this will cause weird behaviors.

Provision folders structure from filesystem to Grafana

If you already store your dashboards using folders in a git repo or on a filesystem, and also you want to have the same folder names in the Grafana menu, you can use foldersFromFilesStructure option.

For example, to replicate these dashboards structure from the filesystem to Grafana,

  1. /etc/dashboards
  2. ├── /server
  3. ├── /common_dashboard.json
  4. └── /network_dashboard.json
  5. └── /application
  6. ├── /requests_dashboard.json
  7. └── /resources_dashboard.json

you need to specify just this short provision configuration file.

  1. apiVersion: 1
  2. providers:
  3. - name: dashboards
  4. type: file
  5. updateIntervalSeconds: 30
  6. options:
  7. path: /etc/dashboards
  8. foldersFromFilesStructure: true

server and application will become new folders in Grafana menu.

Note: folder and folderUid options should be empty or missing to make foldersFromFilesStructure work.

Note: To provision dashboards to the General folder, store them in the root of your path.

Alert Notification Channels

Alert Notification Channels can be provisioned by adding one or more YAML config files in the provisioning/notifiers directory.

Each config file can contain the following top-level fields:

  • notifiers, a list of alert notifications that will be added or updated during start up. If the notification channel already exists, Grafana will update it to match the configuration file.
  • delete_notifiers, a list of alert notifications to be deleted before inserting/updating those in the notifiers list.

Provisioning looks up alert notifications by uid, and will update any existing notification with the provided uid.

By default, exporting a dashboard as JSON will use a sequential identifier to refer to alert notifications. The field uid can be optionally specified to specify a string identifier for the alert name.

  1. {
  2. ...
  3. "alert": {
  4. ...,
  5. "conditions": [...],
  6. "frequency": "24h",
  7. "noDataState": "ok",
  8. "notifications": [
  9. {"uid": "notifier1"},
  10. {"uid": "notifier2"},
  11. ]
  12. }
  13. ...
  14. }

Example Alert Notification Channels Config File

  1. notifiers:
  2. - name: notification-channel-1
  3. type: slack
  4. uid: notifier1
  5. # either
  6. org_id: 2
  7. # or
  8. org_name: Main Org.
  9. is_default: true
  10. send_reminder: true
  11. frequency: 1h
  12. disable_resolve_message: false
  13. # See `Supported Settings` section for settings supported for each
  14. # alert notification type.
  15. settings:
  16. recipient: 'XXX'
  17. uploadImage: true
  18. token: 'xoxb' # legacy setting since Grafana v7.2 (stored non-encrypted)
  19. url: https://slack.com # legacy setting since Grafana v7.2 (stored non-encrypted)
  20. # Secure settings that will be encrypted in the database (supported since Grafana v7.2). See `Supported Settings` section for secure settings supported for each notifier.
  21. secure_settings:
  22. token: 'xoxb'
  23. url: https://slack.com
  24. delete_notifiers:
  25. - name: notification-channel-1
  26. uid: notifier1
  27. # either
  28. org_id: 2
  29. # or
  30. org_name: Main Org.
  31. - name: notification-channel-2
  32. # default org_id: 1

Supported Settings

The following sections detail the supported settings and secure settings for each alert notification type. Secure settings are stored encrypted in the database and you add them to secure_settings in the YAML file instead of settings.

Note: Secure settings is supported since Grafana v7.2.

Alert notification pushover

NameSecure setting
apiTokenyes
userKeyyes
device
priority
okPriority
retry
expire
sound
okSound

Alert notification discord

NameSecure setting
urlyes
avatar_url
content

Alert notification slack

NameSecure setting
urlyes
recipient
username
icon_emoji
icon_url
uploadImage
mentionUsers
mentionGroups
mentionChannel
tokenyes

Alert notification victorops

Name
url
autoResolve

Alert notification kafka

Name
kafkaRestProxy
kafkaTopic

Alert notification LINE

NameSecure setting
tokenyes

Alert notification pagerduty

NameSecure setting
integrationKeyyes
autoResolve

Alert notification sensu

NameSecure setting
url
source
handler
username
passwordyes

Alert notification sensugo

NameSecure setting
url
apikeyyes
entity
check
handler
namespace

Alert notification prometheus-alertmanager

NameSecure setting
url
basicAuthUser
basicAuthPasswordyes

Alert notification teams

Name
url

Alert notification dingding

Name
url

Alert notification email

Name
singleEmail
addresses

Alert notification hipchat

Name
url
apikey
roomid

Alert notification opsgenie

NameSecure setting
apiKeyyes
apiUrl
autoClose
overridePriority
sendTagsAs

Alert notification telegram

NameSecure setting
bottokenyes
chatid
uploadImage

Alert notification threema

NameSecure setting
gateway_id
recipient_id
api_secretyes

Alert notification webhook

NameSecure setting
url
httpMethod
username
passwordyes

Alert notification googlechat

Name
url

Grafana Enterprise

Grafana Enterprise supports provisioning for the following resources: