2 Streaming to external systems

Overview

It is possible to stream item values and events from Zabbix to external systems over HTTP (see protocol details).

This feature currently has experimental status.

The tag filter can be used to stream subsets of item values or events.

Two Zabbix server processes are responsible for data streaming: connector manager and connector worker. A Zabbix internal item zabbix[connector_queue] allows to monitor the count of values enqueued in the connector queue.

Configuration

The following steps are required to configure data streaming to an external system:

1. Have a remote system set up for receiving data from Zabbix.

See the documentation of a simple receiver. The receiver currently logs the received information in events.ndjson and history.ndjson files.

2. Set the required number of connector workers in Zabbix (see StartConnectors in zabbix_server.conf). Restart the Zabbix server.

3. Configure a new connector in Zabbix (Administration -> General -> Connectors) and reload the server cache with “zabbix_server -R config_cache_reload”.

2 Streaming to external systems - 图1

Mandatory fields are marked by an asterisk.

ParameterDescription
NameEnter the connector name.
Data typeSelect the data type: Item values or Events.
URLEnter the receiver URL. User macros are supported.
Tag filterExport only values or events matching the tag filter. If not set, then export everything.
It is possible to include as well as exclude specific tags and tag values. Several conditions can be set. Tag name matching is always case-sensitive.
There are several operators available for each condition:
Exists - include the specified tag names
Equals - include the specified tag names and values (case-sensitive)
Contains - include the specified tag names where the tag values contain the entered string (substring match, case-insensitive)
Does not exist - exclude the specified tag names
Does not equal - exclude the specified tag names and values (case-sensitive)
Does not contain - exclude the specified tag names where the tag values contain the entered string (substring match, case-insensitive)
There are two calculation types for conditions:
And/Or - all conditions must be met, conditions having the same tag name will be grouped by the Or condition
Or - enough if one condition is met
HTTP authenticationSelect the authentication option:
None - no authentication used;
Basic - basic authentication is used;
NTLM - NTLM (Windows NT LAN Manager) authentication is used;
Kerberos - Kerberos authentication is used;
Digest - Digest authentication is used;
Bearer - Bearer authentication is used.
UsernameEnter the user name.
This field is available if HTTP authentication is set to Basic, NTLM, Kerberos, or Digest. User macros are supported.
PasswordEnter the user password.
This field is available if HTTP authentication is set to Basic, NTLM, Kerberos, or Digest. User macros are supported.
Bearer tokenEnter the Bearer token.
This field is available and required if HTTP authentication is set to Bearer. User macros are supported.
Advanced configurationMark this checkbox to display advanced configuration options.
Max records per messageSpecify the maximum number of values or events that can be sent within one message.
Concurrent sessionsSelect the number of sender processes to run for this connector. Up to 100 sessions can be specified; the default value is ‘1’.
AttemptsNumber of attempts for trying to send the data. Up to 5 attempts can be specified; the default value is ‘1’.
TimeoutSpecify the message timeout (1-60 seconds, default 5 seconds).
Time suffixes are supported, e.g. 30s, 1m. User macros are supported.
HTTP proxyYou can specify an HTTP proxy to use, using the format [protocol://][username[:password]@]proxy.example.com[:port].
The optional protocol:// prefix may be used to specify alternative proxy protocols (the protocol prefix support was added in cURL 7.21.7). With no protocol specified, the proxy will be treated as an HTTP proxy.
By default, 1080 port will be used.
If specified, the proxy will overwrite proxy related environment variables like http_proxy, HTTPS_PROXY. If not specified, the proxy will not overwrite proxy-related environment variables. The entered value is passed on “as is”, no sanity checking takes place.
You may also enter a SOCKS proxy address. If you specify the wrong protocol, the connection will fail and the item will become unsupported.
Note that only simple authentication is supported with HTTP proxy.
User macros are supported.
SSL verify peerMark the checkbox to verify the SSL certificate of the web server.
The server certificate will be automatically taken from system-wide certificate authority (CA) location. You can override the location of CA files using Zabbix server or proxy configuration parameter SSLCALocation.
SSL verify hostMark the checkbox to verify that the Common Name field or the Subject Alternate Name field of the web server certificate matches.
This sets the CURLOPT_SSL_VERIFYHOST cURL option.
SSL certificate fileName of the SSL certificate file used for client authentication. The certificate file must be in PEM1 format. If the certificate file contains also the private key, leave the SSL key file field empty. If the key is encrypted, specify the password in SSL key password field. The directory containing this file is specified by Zabbix server or proxy configuration parameter SSLCertLocation. User macros are supported.
SSL key fileName of the SSL private key file used for client authentication. The private key file must be in PEM1 format. The directory containing this file is specified by Zabbix server or proxy configuration parameter SSLKeyLocation. User macros are supported.
SSL key passwordSSL private key file password. User macros are supported.
DescriptionEnter the connector description.
EnabledMark the checkbox to enable the connector.

Protocol

Communication between the server and the receiver is done over HTTP using REST API, NDJSON, “Content-Type: application/x-ndjson”.

For more details, see newline-delimited JSON export protocol.

Server request

Example of sending item values:

  1. POST /v1/history HTTP/1.1
  2. Host: localhost:8080
  3. Accept: */*
  4. Accept-Encoding: deflate, gzip, br, zstd
  5. Content-Length: 628
  6. Content-Type: application/x-ndjson
  7. {"host":{"host":"Zabbix server","name":"Zabbix server"},"groups":["Zabbix servers"],"item_tags":[{"tag":"foo","value":"test"}],"itemid":44457,"name":"foo","clock":1673454303,"ns":800155804,"value":0,"type":3}
  8. {"host":{"host":"Zabbix server","name":"Zabbix server"},"groups":["Zabbix servers"],"item_tags":[{"tag":"foo","value":"test"}],"itemid":44457,"name":"foo","clock":1673454303,"ns":832290669,"value":1,"type":3}
  9. {"host":{"host":"Zabbix server","name":"Zabbix server"},"groups":["Zabbix servers"],"item_tags":[{"tag":"bar","value":"test"}],"itemid":44458,"name":"bar","clock":1673454303,"ns":867770366,"value":123,"type":3}

Example of sending events:

  1. POST /v1/events HTTP/1.1
  2. Host: localhost:8080
  3. Accept: */*
  4. Accept-Encoding: deflate, gzip, br, zstd
  5. Content-Length: 333
  6. Content-Type: application/x-ndjson
  7. {"clock":1673454303,"ns":800155804,"value":1,"eventid":5,"name":"trigger for foo being 0","severity":0,"hosts":[{"host":"Zabbix server","name":"Zabbix server"}],"groups":["Zabbix servers"],"tags":[{"tag":"foo_trig","value":"test"},{"tag":"foo","value":"test"}]}
  8. {"clock":1673454303,"ns":832290669,"value":0,"eventid":6,"p_eventid":5}
Receiver response

The response consists of the HTTP response status code and the JSON payload. The HTTP response status code must be “200” for requests that were handled successfully, other for failed requests.

Example of success:

  1. localhost:8080/v1/history": HTTP/1.1 200 OK
  2. Date: Wed, 11 Jan 2023 16:40:30 GMT
  3. Content-Length: 0

Example with errors:

  1. localhost:8080/v1/history": HTTP/1.1 422 Unprocessable Entity
  2. Content-Type: application/json
  3. Date: Wed, 11 Jan 2023 17:07:36 GMT
  4. Content-Length: 55
  5. {"error":"invalid character '{' after top-level value"}