Mapping Rules

Mapping rules are used to configure the storage policy for metrics. The storage policy determines how long to store metrics for and at what resolution to keep them at. For example, a storage policy of 1m:48h tells M3 to keep the metrics for 48hrs at a 1min resolution. Mapping rules can be configured in the m3coordinator configuration file under the downsample > rules > mappingRules stanza. We will use the following as an example.

  1. downsample:
  2.   rules:
  3.     mappingRules:
  4.       - name: "mysql metrics"
  5.         filter: "app:mysql*"
  6.         aggregations: ["Last"]
  7.         storagePolicies:
  8.           - resolution: 1m
  9.             retention: 48h
  10.       - name: "nginx metrics"
  11.         filter: "app:nginx*"
  12.         aggregations: ["Last"]
  13.         storagePolicies:
  14.           - resolution: 30s
  15.             retention: 24h
  16.           - resolution: 1m
  17.             retention: 48h

Here, we have two mapping rules configured – one for mysql metrics and one for nginx metrics. The filter determines what metrics each rule applies to. The mysql metrics rule will apply to any metrics where the app tag contains mysql* as the value (* being a wildcard). Similarly, the nginx metrics rule will apply to all metrics where the app tag contains nginx* as the value.

The aggregations field determines what functions to apply to the datapoints within a resolution tile. For example, if an application emits a metric every 10sec and the resolution for that metrics’s storage policy is 1min, M3 will need to combine 6 datapoints. If the aggregations policy is Last, M3 will take the last value in that 1min bucket. aggregations can be one of the following:

  1. Last
  2. Min
  3. Max
  4. Mean
  5. Median
  6. Count
  7. Sum
  8. SumSq
  9. Stdev
  10. P10
  11. P20
  12. P30
  13. P40
  14. P50
  15. P60
  16. P70
  17. P80
  18. P90
  19. P95
  20. P99
  21. P999
  22. P9999

Lastly, the storagePolicies field determines which namespaces to store the metrics in. For example, the mysql metrics will be sent to the 1m:48h namespace, while the nginx metrics will be sent to both the 1m:48h and 30s:24h namespaces.

Note: the namespaces listed under the storagePolicies stanza must exist in M3DB.

Rollup Rules

Rollup rules are used to rollup metrics and aggregate in different ways by arbitrary dimensions before they are stored.

Here’s an example of creating a new monotonic counter called http_request_rollup_no_pod_bucket from a set of histogram metrics originally called http_request_bucket:

  1. downsample:
  2.   rules:
  3.     rollupRules:
  4.       - name: "http_request latency by route and git_sha without pod"
  5.         filter: "__name__:http_request_bucket k8s_pod:* le:* git_sha:* route:*"
  6.         transforms:
  7.         - transform:
  8.             type: "Increase"
  9.         - rollup:
  10.             metricName: "http_request_rollup_no_pod_bucket"
  11.             groupBy: ["le", "git_sha", "route", "status_code", "region"]
  12.             aggregations: ["Sum"]
  13.         - transform:
  14.             type: "Add"
  15.         storagePolicies:
  16.         - resolution: 30s
  17.           retention: 720h

Note: only metrics that contain all of the group_by tags will be rolled up. For example, in the above config, only http_request_bucket metrics that have all of the group_by labels present will be rolled up into the new metric http_request_rollup_no_pod_bucket.

While the above example can be used to create a new rolled up metric, often times the goal of rollup rules is to eliminate the underlaying, raw metrics. In order to do this, a mappingRule will need to be added like in the following example (using the metric above as an example) with drop set to true. Additionally, if all of the underlaying metrics are being dropped, there is no need to change the metric name (e.g. in the rollupRule, the metricName field can be equal to the existing metric) – see below for an example.

  1. downsample:
  2.   rules:
  3.     mappingRules:
  4.       - name: "http_request latency by route and git_sha drop raw"
  5.         filter: "__name__:http_request_bucket k8s_pod:* le:* git_sha:* route:*"
  6.         drop: true
  7.     rollupRules:
  8.       - name: "http_request latency by route and git_sha without pod"
  9.         filter: "__name__:http_request_bucket k8s_pod:* le:* git_sha:* route:*"
  10.         transforms:
  11.         - transform:
  12.             type: "Increase"
  13.         - rollup:
  14.             metricName: "http_request_bucket" # metric name doesn't change
  15.             groupBy: ["le", "git_sha", "route", "status_code", "region"]
  16.             aggregations: ["Sum"]
  17.         - transform:
  18.             type: "Add"
  19.         storagePolicies:
  20.         - resolution: 30s
  21.           retention: 720h

Note: In order to store rolled up metrics in an unaggregated namespace, the namespace’s aggregationOptions must have a matching aggregation. For example, if in the above rule, the 720h namespace under storagePolicies is unaggregated, the aggregationOptions for that namespace should resemble the following:

  1. "aggregationOptions": {
  2.   "aggregations": [
  3.     {
  4.       "aggregated": false
  5.     },
  6.     {
  7.       "aggregated": true,
  8.       "attributes": {
  9.         "resolutionDuration": "30s",
  10.         "downsampleOptions": { "all": false }
  11.       }
  12.     }
  13.   ]
  14. }