Alerting with Nagios

OpenTSDB is great, but it’s not (yet) a full monitoring platform. Now that you have a bunch of metrics in OpenTSDB, you want to start sending alerts when thresholds are getting too high. It’s easy!

In the tools directory is a Python script check_tsd. This script queries OpenTSDB and returns Nagios compatible output that gives you OK/WARNING/CRITICAL state.

Parameters

  1. Options:
  2.   -h, --help            show this help message and exit
  3.   -H HOST, --host=HOST  Hostname to use to connect to the TSD.
  4.   -p PORT, --port=PORT  Port to connect to the TSD instance on.
  5.   -m METRIC, --metric=METRIC
  6.                         Metric to query.
  7.   -t TAG, --tag=TAG     Tags to filter the metric on.
  8.   -d SECONDS, --duration=SECONDS
  9.                         How far back to look for data. Default 600s.
  10.   -D METHOD, --downsample=METHOD
  11.                         Downsample function, e.g. one of avg, min, sum, max ... etc
  12.   -W SECONDS, --downsample-window=SECONDS
  13.                         Window size over which to downsample.
  14.   -a METHOD, --aggregator=METHOD
  15.                         Aggregation method: avg, min, sum (default), max .. etc
  16.   -x METHOD, --method=METHOD
  17.                         Comparison method: gt, ge, lt, le, eq, ne.
  18.   -r, --rate            Use rate value as comparison operand.
  19.   -w THRESHOLD, --warning=THRESHOLD
  20.                         Threshold for warning.  Uses the comparison method.
  21.   -c THRESHOLD, --critical=THRESHOLD
  22.                         Threshold for critical.  Uses the comparison method.
  23.   -v, --verbose         Be more verbose.
  24.   -T SECONDS, --timeout=SECONDS
  25.                         How long to wait for the response from TSD.
  26.   -E, --no-result-ok    Return OK when TSD query returns no result.
  27.   -I SECONDS, --ignore-recent=SECONDS
  28.                         Ignore data points that are that are that recent.
  29.   -P PERCENT, --percent-over=PERCENT
  30.                         Only alarm if PERCENT of the data points violate the
  31.                         threshold.
  32.   -N UTC, --now=UTC     Set unix timestamp for "now", for testing
  33.   -S, --ssl             Make queries to OpenTSDB via SSL (https)

For a complete list of downsample & aggregation modes, see http://opentsdb.net/docs/build/html/user_guide/query/aggregators.html#available-aggregators

Nagios Setup

Drop the script into your Nagios path and set up a command like this:

  1. define command{
  2.         command_name check_tsd
  3.         command_line $USER1$/check_tsd -H $HOSTADDRESS$ $ARG1$
  4. }

Then define a host in nagios for your TSD server(s). You can give it a check_command that is guaranteed to always return something if the backend is healthy.

  1. define host{
  2.         host_name               tsd
  3.         address                 tsd
  4.         check_command           check_tsd!-d 60 -m rate:tsd.rpc.received -t type=put -x lt -c 1
  5.         [...]
  6. }

Then define some service checks for the things you want to monitor.

  1. define service{
  2.         host_name                       tsd
  3.         service_description             Apache too many internal errors
  4.         check_command                   check_tsd!-d 300 -m rate:apache.stats.hits -t status=500 -w 1 -c 2
  5.         [...]
  6. }

Testing

If you have want to test your parameters against some specific point in time, you can use the --now <UTC> parameter to specify an explicit unix timestamp which is used as the current timestamp instead of the actual current time. If set, the script will fetch data starting at UTC - duration, ending at UTC.

To see the values retreived, and potentially ignored (due to duration), use the --verbose option.