Monitoring Replication Follower

Note: this recipe is working with ArangoDB 2.5, you need a collectd curl_json plugin with correct boolean type mapping.

Problem

How to monitor the Follower status using the collectd curl_JSON plugin.

Solution

Since arangodb reports the replication status in JSON, integrating it with the collectd curl_JSON plugin should be an easy exercise. However, only very recent versions of collectd will handle boolean flags correctly.

Our test Leader/Follower setup runs with the Leader listening on tcp://127.0.0.1:8529 and the Follower (which we query) listening on tcp://127.0.0.1:8530. They replicate a database by the name testDatabase.

Since replication appliers are active per database and our example doesn’t use the default _system, we need to specify its name in the URL like this: _db/testDatabase.

We need to parse a document from a request like this:

  1. curl --dump - http://localhost:8530/_db/testDatabase/_api/replication/applier-state

If the replication is not running the document will look like that:

  1. {
  2. "state": {
  3. "running": false,
  4. "lastAppliedContinuousTick": null,
  5. "lastProcessedContinuousTick": null,
  6. "lastAvailableContinuousTick": null,
  7. "safeResumeTick": null,
  8. "progress": {
  9. "time": "2015-11-02T13:24:07Z",
  10. "message": "applier shut down",
  11. "failedConnects": 0
  12. },
  13. "totalRequests": 1,
  14. "totalFailedConnects": 0,
  15. "totalEvents": 0,
  16. "totalOperationsExcluded": 0,
  17. "lastError": {
  18. "time": "2015-11-02T13:24:07Z",
  19. "errorMessage": "no start tick",
  20. "errorNum": 1413
  21. },
  22. "time": "2015-11-02T13:31:53Z"
  23. },
  24. "server": {
  25. "version": "2.7.0",
  26. "serverId": "175584498800385"
  27. },
  28. "endpoint": "tcp://127.0.0.1:8529",
  29. "database": "testDatabase"
  30. }

A running replication will return something like this:

  1. {
  2. "state": {
  3. "running": true,
  4. "lastAppliedContinuousTick": "1150610894145",
  5. "lastProcessedContinuousTick": "1150610894145",
  6. "lastAvailableContinuousTick": "1151639153985",
  7. "safeResumeTick": "1150610894145",
  8. "progress": {
  9. "time": "2015-11-02T13:49:56Z",
  10. "message": "fetching leader log from tick 1150610894145",
  11. "failedConnects": 0
  12. },
  13. "totalRequests": 12,
  14. "totalFailedConnects": 0,
  15. "totalEvents": 2,
  16. "totalOperationsExcluded": 0,
  17. "lastError": {
  18. "errorNum": 0
  19. },
  20. "time": "2015-11-02T13:49:57Z"
  21. },
  22. "server": {
  23. "version": "2.7.0",
  24. "serverId": "175584498800385"
  25. },
  26. "endpoint": "tcp://127.0.0.1:8529",
  27. "database": "testDatabase"
  28. }

We create a simple collectd configuration in /etc/collectd/collectd.conf.d/follower_testDatabase.conf that matches our API:

  1. TypesDB "/etc/collectd/collectd.conf.d/followerstate_types.db"
  2. <Plugin curl_json>
  3. # Adjust the URL so collectd can reach your arangod Follower instance:
  4. <URL "http://localhost:8530/_db/testDatabase/_api/replication/applier-state">
  5. # Set your authentication to that database here:
  6. # User "foo"
  7. # Password "bar"
  8. <Key "state/running">
  9. Type "boolean"
  10. </Key>
  11. <Key "state/totalOperationsExcluded">
  12. Type "counter"
  13. </Key>
  14. <Key "state/totalRequests">
  15. Type "counter"
  16. </Key>
  17. <Key "state/totalFailedConnects">
  18. Type "counter"
  19. </Key>
  20. </URL>
  21. </Plugin>

To get nice metric names, we specify our own types.db file in /etc/collectd/collectd.conf.d/followerstate_types.db:

  1. boolean value:ABSOLUTE:0:1

So, basically state/running will give you 0/1 if its (not / ) running through the collectd monitor.