taosAdapter

taosAdapter is a TDengine companion tool that acts as a bridge and adapter between TDengine clusters and applications. It provides an easy-to-use and efficient way to ingest data directly from data collection agent software such as Telegraf, StatsD, collectd, etc. It also provides an InfluxDB/OpenTSDB compatible data ingestion interface that allows InfluxDB/OpenTSDB applications to be seamlessly ported to TDengine.

taosAdapter provides the following features.

  • RESTful interface
  • InfluxDB v1 compliant write interface
  • OpenTSDB JSON and telnet format writes compatible
  • Seamless connection to Telegraf
  • Seamless connection to collectd
  • Seamless connection to StatsD
  • Supports Prometheus remote_read and remote_write
  • Get table’s VGroup ID

taosAdapter architecture diagram

TDengine Database taosAdapter Architecture

taosAdapter Deployment Method

Install taosAdapter

If you use the TDengine server, you don’t need additional steps to install taosAdapter. You can download taosAdapter from TDengine 3.0 released versions to download the TDengine server installation package. If you need to deploy taosAdapter separately on another server other than the TDengine server, you should install the full TDengine server package on that server to install taosAdapter. If you need to build taosAdapter from source code, you can refer to the Building taosAdapter documentation.

Start/Stop taosAdapter

On Linux systems, the taosAdapter service is managed by systemd by default. You can use the command systemctl start taosadapter to start the taosAdapter service and use the command systemctl stop taosadapter to stop the taosAdapter service.

Remove taosAdapter

Use the command rmtaos to remove the TDengine server software if you use tar.gz package. If you installed using a .deb or .rpm package, use the corresponding command, for your package manager, like apt or rpm to remove the TDengine server, including taosAdapter.

Upgrade taosAdapter

taosAdapter and TDengine server need to use the same version. Please upgrade the taosAdapter by upgrading the TDengine server. You need to upgrade the taosAdapter deployed separately from TDengine server by upgrading the TDengine server on the deployed server.

taosAdapter parameter list

taosAdapter is configurable via command-line arguments, environment variables and configuration files. The default configuration file is /etc/taos/taosadapter.toml on Linux.

Command-line arguments take precedence over environment variables over configuration files. The command-line usage is arg=val, e.g., taosadapter -p=30000 —debug=true. The detailed list is as follows:

  1. Usage of taosAdapter:
  2. --collectd.db string collectd db name. Env "TAOS_ADAPTER_COLLECTD_DB" (default "collectd")
  3. --collectd.enable enable collectd. Env "TAOS_ADAPTER_COLLECTD_ENABLE" (default true)
  4. --collectd.password string collectd password. Env "TAOS_ADAPTER_COLLECTD_PASSWORD" (default "taosdata")
  5. --collectd.port int collectd server port. Env "TAOS_ADAPTER_COLLECTD_PORT" (default 6045)
  6. --collectd.user string collectd user. Env "TAOS_ADAPTER_COLLECTD_USER" (default "root")
  7. --collectd.worker int collectd write worker. Env "TAOS_ADAPTER_COLLECTD_WORKER" (default 10)
  8. --collectd.ttl int collectd data ttl. Env "TAOS_ADAPTER_COLLECTD_TTL" (default 0, means no ttl)
  9. -c, --config string config path default /etc/taos/taosadapter.toml
  10. --cors.allowAllOrigins cors allow all origins. Env "TAOS_ADAPTER_CORS_ALLOW_ALL_ORIGINS" (default true)
  11. --cors.allowCredentials cors allow credentials. Env "TAOS_ADAPTER_CORS_ALLOW_Credentials"
  12. --cors.allowHeaders stringArray cors allow HEADERS. Env "TAOS_ADAPTER_ALLOW_HEADERS"
  13. --cors.allowOrigins stringArray cors allow origins. Env "TAOS_ADAPTER_ALLOW_ORIGINS"
  14. --cors.allowWebSockets cors allow WebSockets. Env "TAOS_ADAPTER_CORS_ALLOW_WebSockets"
  15. --cors.exposeHeaders stringArray cors expose headers. Env "TAOS_ADAPTER_Expose_Headers"
  16. --debug enable debug mode. Env "TAOS_ADAPTER_DEBUG"
  17. --help Print this help message and exit
  18. --influxdb.enable enable influxdb. Env "TAOS_ADAPTER_INFLUXDB_ENABLE" (default true)
  19. --log.enableRecordHttpSql whether to record http sql. Env "TAOS_ADAPTER_LOG_ENABLE_RECORD_HTTP_SQL"
  20. --log.path string log path. Env "TAOS_ADAPTER_LOG_PATH" (default "/var/log/taos")
  21. --log.rotationCount uint log rotation count. Env "TAOS_ADAPTER_LOG_ROTATION_COUNT" (default 30)
  22. --log.rotationSize string log rotation size(KB MB GB), must be a positive integer. Env "TAOS_ADAPTER_LOG_ROTATION_SIZE" (default "1GB")
  23. --log.rotationTime duration log rotation time. Env "TAOS_ADAPTER_LOG_ROTATION_TIME" (default 24h0m0s)
  24. --log.sqlRotationCount uint record sql log rotation count. Env "TAOS_ADAPTER_LOG_SQL_ROTATION_COUNT" (default 2)
  25. --log.sqlRotationSize string record sql log rotation size(KB MB GB), must be a positive integer. Env "TAOS_ADAPTER_LOG_SQL_ROTATION_SIZE" (default "1GB")
  26. --log.sqlRotationTime duration record sql log rotation time. Env "TAOS_ADAPTER_LOG_SQL_ROTATION_TIME" (default 24h0m0s)
  27. --logLevel string log level (panic fatal error warn warning info debug trace). Env "TAOS_ADAPTER_LOG_LEVEL" (default "info")
  28. --monitor.collectDuration duration Set monitor duration. Env "TAOS_MONITOR_COLLECT_DURATION" (default 3s)
  29. --monitor.identity string The identity of the current instance, or 'hostname:port' if it is empty. Env "TAOS_MONITOR_IDENTITY"
  30. --monitor.incgroup Whether running in cgroup. Env "TAOS_MONITOR_INCGROUP"
  31. --monitor.password string TDengine password. Env "TAOS_MONITOR_PASSWORD" (default "taosdata") --monitor.pauseAllMemoryThreshold float Memory percentage threshold for pause all. Env "TAOS_MONITOR_PAUSE_ALL_MEMORY_THRESHOLD" (default 80)
  32. --monitor.pauseQueryMemoryThreshold float Memory percentage threshold for pause query. Env "TAOS_MONITOR_PAUSE_QUERY_MEMORY_THRESHOLD" (default 70)
  33. --monitor.user string TDengine user. Env "TAOS_MONITOR_USER" (default "root")
  34. --monitor.writeInterval duration Set write to TDengine interval. Env "TAOS_MONITOR_WRITE_INTERVAL" (default 30s)
  35. --monitor.writeToTD Whether write metrics to TDengine. Env "TAOS_MONITOR_WRITE_TO_TD"
  36. --node_exporter.caCertFile string node_exporter ca cert file path. Env "TAOS_ADAPTER_NODE_EXPORTER_CA_CERT_FILE"
  37. --node_exporter.certFile string node_exporter cert file path. Env "TAOS_ADAPTER_NODE_EXPORTER_CERT_FILE"
  38. --node_exporter.db string node_exporter db name. Env "TAOS_ADAPTER_NODE_EXPORTER_DB" (default "node_exporter")
  39. --node_exporter.enable enable node_exporter. Env "TAOS_ADAPTER_NODE_EXPORTER_ENABLE"
  40. --node_exporter.gatherDuration duration node_exporter gather duration. Env "TAOS_ADAPTER_NODE_EXPORTER_GATHER_DURATION" (default 5s)
  41. --node_exporter.httpBearerTokenString string node_exporter http bearer token. Env "TAOS_ADAPTER_NODE_EXPORTER_HTTP_BEARER_TOKEN_STRING"
  42. --node_exporter.httpPassword string node_exporter http password. Env "TAOS_ADAPTER_NODE_EXPORTER_HTTP_PASSWORD"
  43. --node_exporter.httpUsername string node_exporter http username. Env "TAOS_ADAPTER_NODE_EXPORTER_HTTP_USERNAME"
  44. --node_exporter.insecureSkipVerify node_exporter skip ssl check. Env "TAOS_ADAPTER_NODE_EXPORTER_INSECURE_SKIP_VERIFY" (default true)
  45. --node_exporter.keyFile string node_exporter cert key file path. Env "TAOS_ADAPTER_NODE_EXPORTER_KEY_FILE"
  46. --node_exporter.password string node_exporter password. Env "TAOS_ADAPTER_NODE_EXPORTER_PASSWORD" (default "taosdata")
  47. --node_exporter.responseTimeout duration node_exporter response timeout. Env "TAOS_ADAPTER_NODE_EXPORTER_RESPONSE_TIMEOUT" (default 5s)
  48. --node_exporter.urls strings node_exporter urls. Env "TAOS_ADAPTER_NODE_EXPORTER_URLS" (default [http://localhost:9100])
  49. --node_exporter.user string node_exporter user. Env "TAOS_ADAPTER_NODE_EXPORTER_USER" (default "root")
  50. --node_exporter.ttl int node_exporter data ttl. Env "TAOS_ADAPTER_NODE_EXPORTER_TTL"(default 0, means no ttl)
  51. --opentsdb.enable enable opentsdb. Env "TAOS_ADAPTER_OPENTSDB_ENABLE" (default true)
  52. --opentsdb_telnet.batchSize int opentsdb_telnet batch size. Env "TAOS_ADAPTER_OPENTSDB_TELNET_BATCH_SIZE" (default 1)
  53. --opentsdb_telnet.dbs strings opentsdb_telnet db names. Env "TAOS_ADAPTER_OPENTSDB_TELNET_DBS" (default [opentsdb_telnet,collectd_tsdb,icinga2_tsdb,tcollector_tsdb])
  54. --opentsdb_telnet.enable enable opentsdb telnet,warning: without auth info(default false). Env "TAOS_ADAPTER_OPENTSDB_TELNET_ENABLE"
  55. --opentsdb_telnet.flushInterval duration opentsdb_telnet flush interval (0s means not valid) . Env "TAOS_ADAPTER_OPENTSDB_TELNET_FLUSH_INTERVAL"
  56. --opentsdb_telnet.maxTCPConnections int max tcp connections. Env "TAOS_ADAPTER_OPENTSDB_TELNET_MAX_TCP_CONNECTIONS" (default 250)
  57. --opentsdb_telnet.password string opentsdb_telnet password. Env "TAOS_ADAPTER_OPENTSDB_TELNET_PASSWORD" (default "taosdata")
  58. --opentsdb_telnet.ports ints opentsdb telnet tcp port. Env "TAOS_ADAPTER_OPENTSDB_TELNET_PORTS" (default [6046,6047,6048,6049])
  59. --opentsdb_telnet.tcpKeepAlive enable tcp keep alive. Env "TAOS_ADAPTER_OPENTSDB_TELNET_TCP_KEEP_ALIVE"
  60. --opentsdb_telnet.user string opentsdb_telnet user. Env "TAOS_ADAPTER_OPENTSDB_TELNET_USER" (default "root")
  61. --opentsdb_telnet.ttl int opentsdb_telnet data ttl. Env "TAOS_ADAPTER_OPENTSDB_TELNET_TTL"(default 0, means no ttl)
  62. --pool.idleTimeout duration Set idle connection timeout. Env "TAOS_ADAPTER_POOL_IDLE_TIMEOUT" (default 1h0m0s)
  63. --pool.maxConnect int max connections to taosd. Env "TAOS_ADAPTER_POOL_MAX_CONNECT" (default 4000)
  64. --pool.maxIdle int max idle connections to taosd. Env "TAOS_ADAPTER_POOL_MAX_IDLE" (default 4000)
  65. -P, --port int http port. Env "TAOS_ADAPTER_PORT" (default 6041)
  66. --prometheus.enable enable prometheus. Env "TAOS_ADAPTER_PROMETHEUS_ENABLE" (default true)
  67. --restfulRowLimit int restful returns the maximum number of rows (-1 means no limit). Env "TAOS_ADAPTER_RESTFUL_ROW_LIMIT" (default -1)
  68. --statsd.allowPendingMessages int statsd allow pending messages. Env "TAOS_ADAPTER_STATSD_ALLOW_PENDING_MESSAGES" (default 50000)
  69. --statsd.db string statsd db name. Env "TAOS_ADAPTER_STATSD_DB" (default "statsd")
  70. --statsd.deleteCounters statsd delete counter cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_COUNTERS" (default true)
  71. --statsd.deleteGauges statsd delete gauge cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_GAUGES" (default true)
  72. --statsd.deleteSets statsd delete set cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_SETS" (default true)
  73. --statsd.deleteTimings statsd delete timing cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_TIMINGS" (default true)
  74. --statsd.enable enable statsd. Env "TAOS_ADAPTER_STATSD_ENABLE" (default true)
  75. --statsd.gatherInterval duration statsd gather interval. Env "TAOS_ADAPTER_STATSD_GATHER_INTERVAL" (default 5s)
  76. --statsd.maxTCPConnections int statsd max tcp connections. Env "TAOS_ADAPTER_STATSD_MAX_TCP_CONNECTIONS" (default 250)
  77. --statsd.password string statsd password. Env "TAOS_ADAPTER_STATSD_PASSWORD" (default "taosdata")
  78. --statsd.port int statsd server port. Env "TAOS_ADAPTER_STATSD_PORT" (default 6044)
  79. --statsd.protocol string statsd protocol [tcp or udp]. Env "TAOS_ADAPTER_STATSD_PROTOCOL" (default "udp")
  80. --statsd.tcpKeepAlive enable tcp keep alive. Env "TAOS_ADAPTER_STATSD_TCP_KEEP_ALIVE"
  81. --statsd.user string statsd user. Env "TAOS_ADAPTER_STATSD_USER" (default "root")
  82. --statsd.worker int statsd write worker. Env "TAOS_ADAPTER_STATSD_WORKER" (default 10)
  83. --statsd.ttl int statsd data ttl. Env "TAOS_ADAPTER_STATSD_TTL" (default 0, means no ttl)
  84. --taosConfigDir string load taos client config path. Env "TAOS_ADAPTER_TAOS_CONFIG_FILE"
  85. --version Print the version and exit

Note: Please set the following Cross-Origin Resource Sharing (CORS) parameters according to the actual situation when using a browser for interface calls.

  1. AllowAllOrigins
  2. AllowOrigins
  3. AllowHeaders
  4. ExposeHeaders
  5. AllowCredentials
  6. AllowWebSockets

You do not need to care about these configurations if you do not make interface calls through the browser.

For details on the CORS protocol, please refer to: https://www.w3.org/wiki/CORS_Enabled or https://developer.mozilla.org/zh-CN/docs/Web/HTTP/CORS.

See example/config/taosadapter.toml for sample configuration files.

Feature List

Interfaces

TDengine RESTful interface

You can use any client that supports the http protocol to write data to or query data from TDengine by accessing the REST interface address http://<fqdn>:6041/rest/sql. See the official documentation for details.

InfluxDB

You can use any client that supports the http protocol to access the RESTful interface address http://<fqdn>:6041/<APIEndPoint> to write data in InfluxDB compatible format to TDengine. The EndPoint is as follows:

  1. /influxdb/v1/write

Support InfluxDB query parameters as follows.

  • db Specifies the database name used by TDengine
  • precision The time precision used by TDengine
  • u TDengine user name
  • p TDengine password
  • ttl The time to live of automatically created sub-table. This value cannot be updated. TDengine will use the ttl value of the first data of sub-table to create sub-table. For more information, please refer Create Table

Note: InfluxDB token authorization is not supported at present. Only Basic authorization and query parameter validation are supported. Example: curl —request POST http://127.0.0.1:6041/influxdb/v1/write?db=test —user “root:taosdata” —data-binary “measurement,host=host1 field1=2i,field2=2.0 1577836800000000000”

OpenTSDB

You can use any client that supports the http protocol to access the RESTful interface address http://<fqdn>:6041/<APIEndPoint> to write data in OpenTSDB compatible format to TDengine. The EndPoint is as follows:

  1. /opentsdb/v1/put/json/<db>
  2. /opentsdb/v1/put/telnet/<db>

collectd

Configuring taosAdapter

To configure taosAdapter to receive collectd data.

  • Enable the configuration item in the taosAdapter configuration file (default location is /etc/taos/taosadapter.toml)
  1. ...
  2. [opentsdb_telnet]
  3. enable = true
  4. maxTCPConnections = 250
  5. tcpKeepAlive = false
  6. dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
  7. ports = [6046, 6047, 6048, 6049]
  8. user = "root"
  9. password = "taosdata"
  10. ...

The default database name written by taosAdapter is collectd. You can also modify the taosAdapter configuration file dbs entry to specify a different name. user and password are the values configured by the actual TDengine. After changing the configuration file, you need to restart the taosAdapter.

  • You can also enable the taosAdapter to receive collectd data by using the taosAdapter command-line parameters or by setting environment variables.

Configure collectd

collectd collectd uses a plugin mechanism to write the collected monitoring data to different data storage software in various forms. tdengine supports both direct collection plugins and write_tsdb plugins.

Configure the direct collection plugin

Modify the relevant configuration items in the collectd configuration file (default location /etc/collectd/collectd.conf).

  1. LoadPlugin network
  2. <Plugin network>
  3. Server "<taosAdapter's host>" "<port for collectd direct>"
  4. </Plugin>

where <taosAdapter’s host> fills in the server’s domain name or IP address running taosAdapter. <port for collectd direct> fills in the port that taosAdapter uses to receive collectd data (default is 6045).

An example is as follows.

  1. LoadPlugin network
  2. <Plugin network
  3. Server "127.0.0.1" "6045"
  4. </Plugin>

Configure write_tsdb plugin data

Modify the relevant configuration items in the collectd configuration file (default location /etc/collectd/collectd.conf).

  1. LoadPlugin write_tsdb
  2. <Plugin write_tsdb>
  3. <Node>
  4. Host "<taosAdapter's host>"
  5. Port "<port for collectd write_tsdb plugin>"
  6. ...
  7. </Node>
  8. </Plugin>

Where <taosAdapter’s host> is the domain name or IP address of the server running taosAdapter. <port for collectd write_tsdb plugin> Fill in the data that taosAdapter uses to receive the collectd write_tsdb plugin (default is 6047).

  1. LoadPlugin write_tsdb
  2. <Plugin write_tsdb>
  3. <Node>
  4. Host "127.0.0.1"
  5. Port "6047"
  6. HostTags "status=production"
  7. StoreRates false
  8. AlwaysAppendDS false
  9. </Node
  10. </Plugin>

Then restart collectd.

  1. systemctl restart collectd

StatsD

Configuring taosAdapter

To configure taosAdapter to receive StatsD data.

  • Enable the configuration item in the taosAdapter configuration file (default location /etc/taos/taosadapter.toml)
  1. ...
  2. [statsd]
  3. enable = true
  4. port = 6044
  5. db = "statsd"
  6. user = "root"
  7. password = "taosdata"
  8. worker = 10
  9. gatherInterval = "5s"
  10. protocol = "udp"
  11. maxTCPConnections = 250
  12. tcpKeepAlive = false
  13. allowPendingMessages = 50000
  14. deleteCounters = true
  15. deleteGauges = true
  16. deleteSets = true
  17. deleteTimings = true
  18. ...

The default database name written by taosAdapter is statsd. To specify a different name, you can also modify the taosAdapter configuration file db entry. user and password fill in the actual TDengine configuration values. After changing the configuration file, you need to restart the taosAdapter.

  • You can also enable taosAdapter to receive StatsD data by using the taosAdapter command-line parameters or setting environment variables.

Configuring StatsD

To use StatsD, you need to download its source code. Please refer to the example file exampleConfig.js in the root directory of the source download to modify the configuration file. In <taosAdpater’s host>, please fill in the domain name or IP address of the server running taosAdapter, and <port for StatsD>, please fill in the port where taosAdapter receives StatsD data (default is 6044).

  1. backends section add ". /backends/repeater"
  2. Add { host:'<taosAdapter's host>', port: <port for StatsD>} to repeater section

Example configuration file.

  1. {
  2. port: 8125
  3. , backends: [". /backends/repeater"]
  4. , repeater: [{ host: '127.0.0.1', port: 6044}]
  5. }

Start StatsD after adding the following (assuming the config file is modified to config.js)

  1. npm install
  2. node stats.js config.js &

icinga2 OpenTSDB writer

Configuring taosAdapter

To configure taosAdapter to receive icinga2 data.

  • Enable the configuration item in the taosAdapter configuration file (default location /etc/taos/taosadapter.toml)
  1. ...
  2. [opentsdb_telnet]
  3. enable = true
  4. maxTCPConnections = 250
  5. tcpKeepAlive = false
  6. dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
  7. ports = [6046, 6047, 6048, 6049]
  8. user = "root"
  9. password = "taosdata"
  10. ...

The default database name written by the taosAdapter is icinga2. You can also modify the taosAdapter configuration file dbs entry to specify a different name. user and password are the values configured by the actual TDengine. You need to restart the taosAdapter after modification.

  • You can also enable taosAdapter to receive icinga2 data by using the taosAdapter command-line parameters or setting environment variables.

Configure icinga3

  • Enable opentsdb-writer for icinga2 (refer to the link https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer)
  • Modify the configuration file /etc/icinga2/features-enabled/opentsdb.conf by filling in <taosAdapter’s host> as the domain name or IP address of the server running taosAdapter and <port for icinga2> as the corresponding port on which taosAdapter supports receiving icinga2 data (default is 6048)
  1. object OpenTsdbWriter "opentsdb" {
  2. host = "<taosAdapter's host>"
  3. port = <port for icinga2>
  4. }

Example file:

  1. object OpenTsdbWriter "opentsdb" {
  2. host = "127.0.0.1"
  3. port = 6048
  4. }

TCollector

Configuring taosAdapter

To configure taosAdapter to receive TCollector data.

  • Enable the configuration item in the taosAdapter configuration file (default location /etc/taos/taosadapter.toml)
  1. ...
  2. [opentsdb_telnet]
  3. enable = true
  4. maxTCPConnections = 250
  5. tcpKeepAlive = false
  6. dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
  7. ports = [6046, 6047, 6048, 6049]
  8. user = "root"
  9. password = "taosdata"
  10. ...

The taosAdapter writes to the database with the default name tcollector. You can also modify the taosAdapter configuration file dbs entry to specify a different name. Fill in the actual user and password for TDengine. After changing the configuration file, you need to restart the taosAdapter.

  • You can also enable taosAdapter to receive tcollector data by using the taosAdapter command-line parameters or setting environment variables.

Configuring TCollector

To use TCollector, you need to download its source code. Its configuration items are in its source code. Note: TCollector differs significantly from version to version, so here is an example of the latest code for the current master branch (git commit: 37ae920).

Modify the contents of the collectors/etc/config.py and tcollector.py files. Change the address of the OpenTSDB host to the domain name or IP address of the server where taosAdapter is deployed, and change the port to the port on which taosAdapter supports TCollector (default is 6049).

Example of git diff output of source code changes.

  1. index e7e7a1c..ec3e23c 100644
  2. --- a/collectors/etc/config.py
  3. +++ b/collectors/etc/config.py
  4. @@ -59,13 +59,13 @@ def get_defaults():
  5. 'http_password': False,
  6. 'reconnectinterval': 0,
  7. 'http_username': False,
  8. - 'port': 4242,
  9. + 'port': 6049,
  10. 'pidfile': '/var/run/tcollector.pid',
  11. 'http': False,
  12. 'http_api_path': "api/put",
  13. 'tags': [],
  14. 'remove_inactive_collectors': False,
  15. - 'host': '',
  16. + 'host': '127.0.0.1',
  17. 'logfile': '/var/log/tcollector.log',
  18. 'cdir': default_cdir,
  19. 'ssl': False,
  20. diff --git a/tcollector.py b/tcollector.py
  21. index 21f9b23..4c71ba2 100755
  22. --- a/tcollector.py
  23. +++ b/tcollector.py
  24. @@ -64,7 +64,7 @@ ALIVE = True
  25. # exceptions, something is not right and tcollector will shutdown.
  26. # Hopefully some kind of supervising daemon will then restart it.
  27. MAX_UNCAUGHT_EXCEPTIONS = 100
  28. -DEFAULT_PORT = 4242
  29. +DEFAULT_PORT = 6049
  30. MAX_REASONABLE_TIMESTAMP = 2209212000 # Good until Tue 3 Jan 14:00:00 GMT 2040
  31. # How long to wait for datapoints before assuming
  32. # a collector is dead and restarting it
  33. @@ -943,13 +943,13 @@ def parse_cmdline(argv):
  34. 'http_password': False,
  35. 'reconnectinterval': 0,
  36. 'http_username': False,
  37. - 'port': 4242,
  38. + 'port': 6049,
  39. 'pidfile': '/var/run/tcollector.pid',
  40. 'http': False,
  41. 'http_api_path': "api/put",
  42. 'tags': [],
  43. 'remove_inactive_collectors': False,
  44. - 'host': '',
  45. + 'host': '127.0.0.1',
  46. 'logfile': '/var/log/tcollector.log',
  47. 'cdir': default_cdir,
  48. 'ssl': False,

node_exporter

node_export is an exporter of hardware and OS metrics exposed by the *NIX kernel used by Prometheus

  • Enable the taosAdapter configuration node_exporter.enable
  • Set the configuration of the node_exporter
  • Restart taosAdapter

Prometheus

Configuring Prometheus is done by editing the Prometheus configuration file prometheus.yml (default location /etc/prometheus/prometheus.yml).

Configuring third-party database addresses

Point the remote_read url and remote_write url to the domain name or IP address of the server running the taosAdapter service, the REST service port (taosAdapter uses 6041 by default), and the name of the database you want to write to TDengine, and ensure that the corresponding URL form as follows.

  • remote_read url : http://<taosAdapter's host>:<REST service port>/prometheus/v1/remote_read/<database name>
  • remote_write url : http://<taosAdapter's host>:<REST service port>/prometheus/v1/remote_write/<database name>

Configure Basic authentication

  • username: <TDengine’s username>
  • password: <TDengine’s password>
  1. remote_write:
  2. - url: "http://localhost:6041/prometheus/v1/remote_write/prometheus_data"
  3. basic_auth:
  4. username: root
  5. password: taosdata
  6. remote_read:
  7. - url: "http://localhost:6041/prometheus/v1/remote_read/prometheus_data"
  8. basic_auth:
  9. username: root
  10. password: taosdata
  11. remote_timeout: 10s
  12. read_recent: true

Get table’s VGroup ID

You can call http://<fqdn>:6041/rest/vgid?db=<db>&table=<table> to get table’s VGroup ID. For more information about VGroup, please refer to primary-logic-unit.

Memory usage optimization methods

taosAdapter will monitor its memory usage during operation and adjust it with two thresholds. Valid values are integers between 1 to 100, and represent a percentage of the system’s physical memory.

  • pauseQueryMemoryThreshold
  • pauseAllMemoryThreshold

Stops processing query requests when the pauseQueryMemoryThreshold threshold is exceeded.

HTTP response content.

  • code 503
  • body “query memory exceeds threshold”

Stops processing all write and query requests when the pauseAllMemoryThreshold threshold is exceeded.

HTTP response content.

  • code 503
  • body “memory exceeds threshold”

Resume the corresponding function when the memory falls back below the threshold.

Status check interface http://<fqdn>:6041/-/ping

  • Normal returns code 200
  • No parameter If memory exceeds pauseAllMemoryThreshold returns code 503
  • Request parameter action=query returns code 503 if memory exceeds pauseQueryMemoryThreshold or pauseAllMemoryThreshold

Corresponding configuration parameter

  1. monitor.collectDuration monitoring interval environment variable `TAOS_MONITOR_COLLECT_DURATION` (default value 3s)
  2. monitor.incgroup whether to run in cgroup (set to true for running in container) environment variable `TAOS_MONITOR_INCGROUP`
  3. monitor.pauseAllMemoryThreshold memory threshold for no more inserts and queries environment variable `TAOS_MONITOR_PAUSE_ALL_MEMORY_THRESHOLD` (default 80)
  4. monitor.pauseQueryMemoryThreshold memory threshold for no more queries Environment variable `TAOS_MONITOR_PAUSE_QUERY_MEMORY_THRESHOLD` (default 70)

You should adjust this parameter based on your specific application scenario and operation strategy. We recommend using monitoring software to monitor system memory status. The load balancer can also check the taosAdapter running status through this interface.

taosAdapter Monitoring Metrics

taosAdapter collects HTTP-related metrics, CPU percentage, and memory percentage.

HTTP interface

Provides an interface conforming to OpenMetrics.

  1. http://<fqdn>:6041/metrics

Write to TDengine

taosAdapter supports writing the metrics of HTTP monitoring, CPU percentage, and memory percentage to TDengine.

For configuration parameters

Configuration itemsDescriptionDefault values
monitor.collectDurationCPU and memory collection interval3s
monitor.identityThe current taosadapter identifier will be used if not set to hostname:port
monitor.incgroupwhether it is running in a cgroup (set to true for running in a container)false
monitor.writeToTDWhether to write to TDenginefalse
monitor.userTDengine connection usernameroot
monitor.passwordTDengine connection passwordtaosdata
monitor.writeIntervalWrite to TDengine interval30s

Limit the number of results returned

taosAdapter controls the number of results returned by the parameter restfulRowLimit, -1 means no limit, default is no limit.

This parameter controls the number of results returned by the following interfaces:

  • http://<fqdn>:6041/rest/sql
  • http://<fqdn>:6041/prometheus/v1/remote_read/:db

Troubleshooting

You can check the taosAdapter running status with the systemctl status taosadapter command.

You can also adjust the level of the taosAdapter log output by setting the --logLevel parameter or the environment variable TAOS_ADAPTER_LOG_LEVEL. Valid values are: panic, fatal, error, warn, warning, info, debug and trace.

How to migrate from older TDengine versions to taosAdapter

In TDengine server 2.2.x.x or earlier, the TDengine server process (taosd) contains an embedded HTTP service. As mentioned earlier, taosAdapter is a standalone software managed using systemd and has its own process ID. There are some configuration parameters and behaviors that are different between the two. See the following table for details.

#embedded httpdtaosAdaptercomment
1httpEnableRecordSql—logLevel=debug
2httpMaxThreadsn/ataosAdapter Automatically manages thread pools without this parameter
3telegrafUseFieldNumSee the taosAdapter telegraf configuration method
4restfulRowLimitrestfulRowLimitEmbedded httpd outputs 10240 rows of data by default, the maximum allowed is 102400. taosAdapter also provides restfulRowLimit but it is not limited by default. You can configure it according to the actual scenario.
5httpDebugFlagNot applicablehttpdDebugFlag does not work for taosAdapter
6httpDBNameMandatoryN/AtaosAdapter requires the database name to be specified in the URL