Integrate Prometheus with OSM

Prometheus and OSM Integration

To familiarize yourself on how OSM works with Prometheus, try installing a new mesh with sample applications to see which metrics are collected.

  1. Install OSM with its own Prometheus instance:

    1. $ osm install --set osm.deployPrometheus=true,osm.enablePermissiveTrafficPolicy=true
    2. OSM installed successfully in namespace [osm-system] with mesh name [osm]

  2. Create a namespace for sample workloads:

    1. $ kubectl create namespace metrics-demo
    2. namespace/metrics-demo created

  3. Make the new OSM monitor the new namespace:

    1. $ osm namespace add metrics-demo
    2. Namespace [metrics-demo] successfully added to mesh [osm]

  4. Configure OSM’s Prometheus to scrape metrics from the new namespace:

    1. $ osm metrics enable --namespace metrics-demo
    2. Metrics successfully enabled in namespace [metrics-demo]

  5. Install sample applications:

    1. $ kubectl apply -f https://raw.githubusercontent.com/openservicemesh/osm-docs/release-v1.1/manifests/samples/curl/curl.yaml -n metrics-demo
    2. serviceaccount/curl created
    3. deployment.apps/curl created
    4. $ kubectl apply -f https://raw.githubusercontent.com/openservicemesh/osm-docs/release-v1.1/manifests/samples/httpbin/httpbin.yaml -n metrics-demo
    5. serviceaccount/httpbin created
    6. service/httpbin created
    7. deployment.apps/httpbin created

    Ensure the new Pods are Running and all containers are ready:

    1. $ kubectl get pods -n metrics-demo
    2. NAME                       READY   STATUS    RESTARTS   AGE
    3. curl-54ccc6954c-q8s89      2/2     Running   0          95s
    4. httpbin-8484bfdd46-vq98x   2/2     Running   0          72s

  6. Generate traffic:

    The following command makes the curl Pod make about 1 request per second to the httpbin Pod forever:

    1. $ kubectl exec -n metrics-demo -ti "$(kubectl get pod -n metrics-demo -l app=curl -o jsonpath='{.items[0].metadata.name}')" -c curl -- sh -c 'while :; do curl -i httpbin.metrics-demo:14001/status/200; sleep 1; done'
    2. HTTP/1.1 200 OK
    3. server: envoy
    4. date: Tue, 23 Mar 2021 17:27:44 GMT
    5. content-type: text/html; charset=utf-8
    6. access-control-allow-origin: *
    7. access-control-allow-credentials: true
    8. content-length: 0
    9. x-envoy-upstream-service-time: 1
    11. HTTP/1.1 200 OK
    12. server: envoy
    13. date: Tue, 23 Mar 2021 17:27:45 GMT
    14. content-type: text/html; charset=utf-8
    15. access-control-allow-origin: *
    16. access-control-allow-credentials: true
    17. content-length: 0
    18. x-envoy-upstream-service-time: 2
    20. ...

  7. View metrics in Prometheus:

    Forward the Prometheus port:

    1. $ kubectl port-forward -n osm-system $(kubectl get pods -n osm-system -l app=osm-prometheus -o jsonpath='{.items[0].metadata.name}') 7070
    2. Forwarding from 127.0.0.1:7070 -> 7070
    3. Forwarding from [::1]:7070 -> 7070

    Navigate to http://localhost:7070 in a web browser to view the Prometheus UI. The following query shows how many requests per second are being made from the curl pod to the httpbin pod, which should be about 1:

    1. irate(envoy_cluster_upstream_rq_xx{source_service="curl", envoy_cluster_name="metrics-demo/httpbin"}[30s])

    Feel free to explore the other metrics available from within the Prometheus UI.

  8. Cleanup

    Once you are done with the demo resources, clean them up by first deleting the application namespace:

    1. $ kubectl delete ns metrics-demo
    2. namespace "metrics-demo" deleted

    Then, uninstall OSM:

    1. $ osm uninstall mesh
    2. Uninstall OSM [mesh name: osm] ? [y/n]: y
    3. OSM [mesh name: osm] uninstalled

    To remove OSM’s cluster wide resources after uninstallation, run the following command. See the uninstall guide for more context and information.

    1. $ osm uninstall mesh --delete-cluster-wide-resources