Perf counters

The perf counters provide generic internal infrastructure for gauges and counters. The counted values can be both integer and float. There is also an “average” type (normally float) that combines a sum and num counter which can be divided to provide an average.

The intention is that this data will be collected and aggregated by a tool like collectd or statsd and fed into a tool like graphite for graphing and analysis. Also, note the Prometheus Module.

Access

The perf counter data is accessed via the admin socket. For example:

  1. ceph daemon osd.0 perf schema
  2. ceph daemon osd.0 perf dump

Collections

The values are grouped into named collections, normally representing a subsystem or an instance of a subsystem. For example, the internal throttle mechanism reports statistics on how it is throttling, and each instance is named something like:

  1. throttle-msgr_dispatch_throttler-hbserver
  2. throttle-msgr_dispatch_throttler-client
  3. throttle-filestore_bytes
  4. ...

Schema

The perf schema command dumps a json description of which values are available, and what their type is. Each named value as a type bitfield, with the following bits defined.

bitmeaning
1floating point value
2unsigned 64-bit integer value
4average (sum + count pair), where
8counter (vs gauge)

Every value will have either bit 1 or 2 set to indicate the type(float or integer).

If bit 8 is set (counter), the value is monotonically increasing andthe reader may want to subtract off the previously read value to getthe delta during the previous interval.

If bit 4 is set (average), there will be two values to read, a sum anda count. If it is a counter, the average for the previous intervalwould be sum delta (since the previous read) divided by the countdelta. Alternatively, dividing the values outright would provide thelifetime average value. Normally these are used to measure latencies(number of requests and a sum of request latencies), and the averagefor the previous interval is what is interesting.

Instead of interpreting the bit fields, the metric type has avalue of either guage or counter, and the value typeproperty will be one of real, integer, real-integer-pair(for a sum + real count pair), or integer-integer-pair (for asum + integer count pair).

Here is an example of the schema output:

  1. {
  2.   "throttle-bluestore_throttle_bytes": {
  3.       "val": {
  4.           "type": 2,
  5.           "metric_type": "gauge",
  6.           "value_type": "integer",
  7.           "description": "Currently available throttle",
  8.           "nick": ""
  9.       },
  10.       "max": {
  11.           "type": 2,
  12.           "metric_type": "gauge",
  13.           "value_type": "integer",
  14.           "description": "Max value for throttle",
  15.           "nick": ""
  16.       },
  17.       "get_started": {
  18.           "type": 10,
  19.           "metric_type": "counter",
  20.           "value_type": "integer",
  21.           "description": "Number of get calls, increased before wait",
  22.           "nick": ""
  23.       },
  24.       "get": {
  25.           "type": 10,
  26.           "metric_type": "counter",
  27.           "value_type": "integer",
  28.           "description": "Gets",
  29.           "nick": ""
  30.       },
  31.       "get_sum": {
  32.           "type": 10,
  33.           "metric_type": "counter",
  34.           "value_type": "integer",
  35.           "description": "Got data",
  36.           "nick": ""
  37.       },
  38.       "get_or_fail_fail": {
  39.           "type": 10,
  40.           "metric_type": "counter",
  41.           "value_type": "integer",
  42.           "description": "Get blocked during get_or_fail",
  43.           "nick": ""
  44.       },
  45.       "get_or_fail_success": {
  46.           "type": 10,
  47.           "metric_type": "counter",
  48.           "value_type": "integer",
  49.           "description": "Successful get during get_or_fail",
  50.           "nick": ""
  51.       },
  52.       "take": {
  53.           "type": 10,
  54.           "metric_type": "counter",
  55.           "value_type": "integer",
  56.           "description": "Takes",
  57.           "nick": ""
  58.       },
  59.       "take_sum": {
  60.           "type": 10,
  61.           "metric_type": "counter",
  62.           "value_type": "integer",
  63.           "description": "Taken data",
  64.           "nick": ""
  65.       },
  66.       "put": {
  67.           "type": 10,
  68.           "metric_type": "counter",
  69.           "value_type": "integer",
  70.           "description": "Puts",
  71.           "nick": ""
  72.       },
  73.       "put_sum": {
  74.           "type": 10,
  75.           "metric_type": "counter",
  76.           "value_type": "integer",
  77.           "description": "Put data",
  78.           "nick": ""
  79.       },
  80.       "wait": {
  81.           "type": 5,
  82.           "metric_type": "gauge",
  83.           "value_type": "real-integer-pair",
  84.           "description": "Waiting latency",
  85.           "nick": ""
  86.       }
  87. }

Dump

The actual dump is similar to the schema, except that average values are grouped. For example:

  1. {
  2.   "throttle-msgr_dispatch_throttler-hbserver" : {
  3.      "get_or_fail_fail" : 0,
  4.      "get_sum" : 0,
  5.      "max" : 104857600,
  6.      "put" : 0,
  7.      "val" : 0,
  8.      "take" : 0,
  9.      "get_or_fail_success" : 0,
  10.      "wait" : {
  11.         "avgcount" : 0,
  12.         "sum" : 0
  13.      },
  14.      "get" : 0,
  15.      "take_sum" : 0,
  16.      "put_sum" : 0
  17.   },
  18.   "throttle-msgr_dispatch_throttler-client" : {
  19.      "get_or_fail_fail" : 0,
  20.      "get_sum" : 82760,
  21.      "max" : 104857600,
  22.      "put" : 2637,
  23.      "val" : 0,
  24.      "take" : 0,
  25.      "get_or_fail_success" : 0,
  26.      "wait" : {
  27.         "avgcount" : 0,
  28.         "sum" : 0
  29.      },
  30.      "get" : 2637,
  31.      "take_sum" : 0,
  32.      "put_sum" : 82760
  33.   }
  34. }