Creating an M3 Cluster with Kubernetes

Creating an M3 Cluster with Kubernetes

This guide shows you how to create an M3 cluster of 3 nodes, designed to run locally on the same machine. It is designed to show you how M3 and Kubernetes can work together, but not as a production example.

This guide assumes you have read the quickstart, and builds upon the concepts in that guide.

We recommend you use our Kubernetes operator to deploy M3 to a cluster. It is a more streamlined setup that uses custom resource definitions to automatically handle operations such as managing cluster placements.

Prerequisites

A running Kubernetes cluster.
- For local testing, you can use minikube, Docker desktop, or we have a script you can use to start a 3 node cluster with Kind.

Create An etcd Cluster

M3 stores its cluster placements and runtime metadata in etcd and needs a running cluster to communicate with.

We have example services and stateful sets you can use, but feel free to use your own configuration and change any later instructions accordingly.

kubectl apply -f https://raw.githubusercontent.com/m3db/m3db-operator/master/example/etcd/etcd-minikube.yaml

If the etcd cluster is running on your local machine, update your /etc/hosts file to match the domains specified in the etcd --initial-cluster argument. For example to match the StatefulSet declaration in the etcd-minikube.yaml above, that is:

$(minikube ip) etcd-0.etcd
$(minikube ip) etcd-1.etcd
$(minikube ip) etcd-2.etcd

Verify that the cluster is running with something like the Kubernetes dashboard, or the command below:

kubectl exec etcd-0 -- env ETCDCTL_API=3 etcdctl endpoint health

Install the Operator

Install the bundled operator manifests in the current namespace:

kubectl apply -f https://raw.githubusercontent.com/m3db/m3db-operator/master/bundle.yaml

Create an M3 Cluster

The following creates an M3 cluster with 3 replicas of data across 256 shards that connects to the 3 available etcd endpoints.

It creates three isolated groups for nodes, each with one node instance. In a production environment you can use a variety of different options to define how nodes are spread across groups based on factors such as resource capacity, or location.

It creates namespaces in the cluster with the namespaces parameter. You can use M3-provided presets, or define your own. This example creates a namespace with the 10s:2d preset.

The cluster derives pod identity from the podIdentityConfig parameter, which in this case is the UID of the Pod.

kubectl apply -f https://raw.githubusercontent.com/m3db/m3db-operator/master/example/m3db-local.yaml

Verify that the cluster is running with something like the Kubernetes dashboard, or the command below:

kubectl exec simple-cluster-rep2-0 -- curl -sSf localhost:9002/health

Deleting a Cluster

Delete the M3 cluster using kubectl:

kubectl delete m3dbcluster simple-cluster

By default, the operator uses finalizers to delete the placement and namespaces associated with a cluster before the custom resources. If you do not want this behavior, set keepEtcdDataOnDelete to true in the cluster configuration.

Writing and Querying Metrics

Writing Metrics

M3 supports ingesting statsd and Prometheus formatted metrics.

This quickstart focuses on Prometheus metrics which consist of a value, a timestamp, and tags to bring context and meaning to the metric.

You can write metrics using one of two endpoints:

http://localhost:7201/api/v1/prom/remote/write - Write a Prometheus remote write query to M3DB with a binary snappy compressed Prometheus WriteRequest protobuf message.
http://localhost:7201/api/v1/json/write - Write a JSON payload of metrics data. This endpoint is quick for testing purposes but is not as performant for production usage.

For this quickstart, use the http://localhost:7201/api/v1/json/write endpoint to write a tagged metric to M3 with the following data in the request body, all fields are required:

tags: An object of at least one name/value pairs
timestamp: The UNIX timestamp for the data
value: The value for the data, can be of any type

The examples below use __name__ as the name for one of the tags, which is a Prometheus reserved tag that allows you to query metrics using the value of the tag to filter results.

Label names may contain ASCII letters, numbers, underscores, and Unicode characters. They must match the regex [a-zA-Z_][a-zA-Z0-9_]*. Label names beginning with __ are reserved for internal use. Read more in the Prometheus documentation.

Command 1
Command 2
Command 3

#!/bin/bash
curl -X POST http://localhost:7201/api/v1/json/write -d '{
  "tags": 
    {
      "__name__": "third_avenue",
      "city": "new_york",
      "checkout": "1"
    },
    "timestamp": '\"$(date "+%s")\"',
    "value": 3347.26
}'

#!/bin/bash
curl -X POST http://localhost:7201/api/v1/json/write -d '{
  "tags": 
    {
      "__name__": "third_avenue",
      "city": "new_york",
      "checkout": "1"
    },
    "timestamp": '\"$(date "+%s")\"',
    "value": 5347.26
}'

#!/bin/bash
curl -X POST http://localhost:7201/api/v1/json/write -d '{
  "tags": 
    {
      "__name__": "third_avenue",
      "city": "new_york",
      "checkout": "1"
    },
    "timestamp": '\"$(date "+%s")\"',
    "value": 7347.26
}'

Querying metrics

M3 supports three query engines: Prometheus (default), Graphite, and the M3 Query Engine.

This quickstart uses Prometheus as the query engine, and you have access to all the features of PromQL queries.

To query metrics, use the http://localhost:7201/api/v1/query\_range endpoint with the following data in the request body, all fields are required:

query: A PromQL query
start: Timestamp in RFC3339Nano of start range for results
end: Timestamp in RFC3339Nano of end range for results
step: A duration or float of the query resolution, the interval between results in the timespan between start and end.

Below are some examples using the metrics written above.

curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
  -d "query=third_avenue" \
  -d "start=$(date "+%s" -d "45 seconds ago")" \
  -d "end=$( date +%s )" \
  -d "step=5s" | jq .

curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
  -d "query=third_avenue" \
  -d "start=$( date -v -45S +%s )" \
  -d "end=$( date +%s )" \
  -d "step=5s" | jq .

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {
          "__name__": "third_avenue",
          "checkout": "1",
          "city": "new_york"
        },
        "values": [
          [
            1610746220,
            "3347.26"
          ],
          [
            1610746220,
            "5347.26"
          ],
          [
            1610746220,
            "7347.26"
          ]
        ]
      }
    ]
  }
}

Values above a certain number

Linux
macOS/BSD
Output

curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
  -d "query=third_avenue > 6000" \
  -d "start=$(date "+%s" -d "45 seconds ago")" \
  -d "end=$( date +%s )" \
  -d "step=5s" | jq .

curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
  -d "query=third_avenue > 6000" \
  -d "start=$(date -v -45S "+%s")" \
  -d "end=$( date +%s )" \
  -d "step=5s" | jq .

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {
          "__name__": "third_avenue",
          "checkout": "1",
          "city": "new_york"
        },
        "values": [
          [
            1610746220,
            "7347.26"
          ]
        ]
      }
    ]
  }
}