Promtail Scraping (Service Discovery)

File Target Discovery

Promtail discovers locations of log files and extract labels from them throughthe scrape_configs section in the config YAML. The syntax is identical to whatPrometheus uses.

scrape_configs contains one or more entries which are executed for eachdiscovered target (i.e., each container in each new pod running in theinstance):

  1. scrape_configs:
  2.   - job_name: local
  3.     static_configs:
  4.       - ...
  6.   - job_name: kubernetes
  7.     kubernetes_sd_config:
  8.       - ...

If more than one scrape config section matches your logs, you will get duplicateentries as the logs are sent in different streams likely with slightlydifferent labels.

There are different types of labels present in Promtail:

  • Labels starting with __ (two underscores) are internal labels. They usuallycome from dynamic sources like service discovery. Once relabeling is done,they are removed from the label set. To persist internal labels so they’resent to Loki, rename them so they don’t start with __. SeeRelabeling for more information.

  • Labels starting with __meta_kubernetes_pod_label_* are “meta labels” whichare generated based on your Kubernetes pod’s labels.

    For example, if your Kubernetes pod has a label name set to foobar, thenthe scrape_configs section will receive an internal label__meta_kubernetes_pod_label_name with a value set to foobar.

  • Other labels starting with __meta_kubernetes_* exist based on otherKubernetes metadata, such as the namespace of the pod(__meta_kubernetes_namespace) or the name of the container inside the pod(__meta_kubernetes_pod_container_name). Refer tothe Prometheus docsfor the full list of Kubernetes meta labels.

  • The __path__ label is a special label which Promtail uses after discovery tofigure out where the file to read is located. Wildcards are allowed.

  • The label filename is added for every file found in __path__ to ensure theuniqueness of the streams. It is set to the absolute path of the file the linewas read from.

Kubernetes Discovery

Note that while Promtail can utilize the Kubernetes API to discover pods astargets, it can only read log files from pods that are running on the same nodeas the one Promtail is running on. Promtail looks for a __host__ label oneach target and validates that it is set to the same hostname as Promtail’s(using either $HOSTNAME or the hostname reported by the kernel if theenvironment variable is not set).

This means that any time Kubernetes service discovery is used, there must be arelabel_config that creates the intermediate label __host__ from__meta_kubernetes_pod_node_name:

  1. relabel_configs:
  2.   - source_labels: ['__meta_kubernetes_pod_node_name']
  3.     target_label: '__host__'

See Relabeling for more information.

Relabeling

Each scrape_configs entry can contain a relabel_configs stanza.relabel_configs is a list of operations to transform the labels from discoveryinto another form.

A single entry in relabel_configs can also reject targets by doing an action: drop if a label value matches a specified regex. When a target is dropped, theowning scrape_config will not process logs from that particular source.Other scrape_configs without the drop action reading from the same targetmay still use and forward logs from it to Loki.

A common use case of relabel_configs is to transform an internal label suchas __meta_kubernetes_* into an intermediate internal label such as__service__. The intermediate internal label may then be dropped based onvalue or transformed to a final external label, such as __job__.

Examples

  • Drop the target if a label (__service__ in the example) is empty:
    1. - action: drop
    2.   regex: ^$
    3.   source_labels:
    4.   - __service__
  • Drop the target if any of the source_labels contain a value:
    1. - action: drop
    2.   regex: .+
    3.   separator: ''
    4.   source_labels:
    5.   - __meta_kubernetes_pod_label_name
    6.   - __meta_kubernetes_pod_label_app
  • Persist an internal label by renaming it so it will be sent to Loki:
    1. - action: replace
    2.   source_labels:
    3.   - __meta_kubernetes_namespace
    4.   target_label: namespace
  • Persist all Kubernetes pod labels by mapping them, like by mapping __meta_kube__meta_kubernetes_pod_label_foo to foo.
    1. - action: labelmap
    2.   regex: __meta_kubernetes_pod_label_(.+)

Additional reading:

HTTP client options

Promtail uses the Prometheus HTTP client implementation for all calls to Loki.Therefore it can be configured using the clients stanza, where one or moreconnections to Loki can be established:

  1. clients:
  2.   - [ <client_option> ]

Refer to client_config from the PromtailConfiguration reference for all available options.