clickhouse-logger

Description

clickhouse-logger is a plugin which push Log data requests to clickhouse.

Attributes

NameTypeRequirementDefaultValidDescription
endpoint_addrstringrequiredThe clickhouse endpoint.
databasestringrequiredThe DB name to store log.
logtablestringrequiredThe table name.
userstringrequiredclickhouse user.
passwordstringrequiredclickhouse password.
timeoutintegeroptional3[1,…]Time to keep the connection alive after sending a request.
namestringoptional“clickhouse logger”A unique identifier to identity the logger.
ssl_verifybooleanoptionaltrue[true,false]verify ssl.

The plugin supports the use of batch processors to aggregate and process entries(logs/data) in a batch. This avoids frequent data submissions by the plugin, which by default the batch processor submits data every 5 seconds or when the data in the queue reaches 1000. For information or custom batch processor parameter settings, see Batch-Processor configuration section.

How To Enable

The following is an example of how to enable the clickhouse-logger for a specific route.

  1. curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
  2. {
  3.       "plugins": {
  4.             "clickhouse-logger": {
  5.                 "user": "default",
  6.                 "password": "a",
  7.                 "database": "default",
  8.                 "logtable": "test",
  9.                 "endpoint_addr": "http://127.0.0.1:8123"
  10.             }
  11.        },
  12.       "upstream": {
  13.            "type": "roundrobin",
  14.            "nodes": {
  15.                "127.0.0.1:1980": 1
  16.            }
  17.       },
  18.       "uri": "/hello"
  19. }'

Test Plugin

success:

  1. $ curl -i http://127.0.0.1:9080/hello
  2. HTTP/1.1 200 OK
  3. ...
  4. hello, world

Metadata

NameTypeRequirementDefaultValidDescription
log_formatobjectoptional{“host”: “$host”, “@timestamp”: “$time_iso8601”, “client_ip”: “$remote_addr”}Log format declared as key value pair in JSON format. Only string is supported in the value part. If the value starts with $, it means to get APISIX variable or Nginx variable.

Note that the metadata configuration is applied in global scope, which means it will take effect on all Route or Service which use clickhouse-logger plugin.

Example

  1. curl http://127.0.0.1:9080/apisix/admin/plugin_metadata/clickhouse-logger -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
  2. {
  3.     "log_format": {
  4.         "host": "$host",
  5.         "@timestamp": "$time_iso8601",
  6.         "client_ip": "$remote_addr"
  7.     }
  8. }'

create clickhouse log table

  1. CREATE TABLE default.test (
  2.   `host` String,
  3.   `client_ip` String,
  4.   `route_id` String,
  5.   `@timestamp` String,
  6.    PRIMARY KEY(`@timestamp`)
  7. ) ENGINE = MergeTree()

On clickhouse run select * from default.test;, will got below row:

  1. ┌─host──────┬─client_ip─┬─route_id─┬─@timestamp────────────────┐
  2. │ 127.0.0.1 │ 127.0.0.1 │ 1        │ 2022-01-17T10:03:10+08:00 │
  3. └───────────┴───────────┴──────────┴───────────────────────────┘

Disable Plugin

Remove the corresponding json configuration in the plugin configuration to disable the clickhouse-logger. APISIX plugins are hot-reloaded, therefore no need to restart APISIX.

  1. $ curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
  2. {
  3.     "uri": "/hello",
  4.     "plugins": {},
  5.     "upstream": {
  6.         "type": "roundrobin",
  7.         "nodes": {
  8.             "127.0.0.1:1980": 1
  9.         }
  10.     }
  11. }'