Operators
Binary operators
Prometheus's query language supports basic logical and arithmetic operators.For operations between two instant vectors, the matching behaviorcan be modified.
Arithmetic binary operators
The following binary arithmetic operators exist in Prometheus:
+
(addition)-
(subtraction)*
(multiplication)/
(division)%
(modulo)^
(power/exponentiation)Binary arithmetic operators are defined between scalar/scalar, vector/scalar,and vector/vector value pairs.
Between two scalars, the behavior is obvious: they evaluate to anotherscalar that is the result of the operator applied to both scalar operands.
Between an instant vector and a scalar, the operator is applied to thevalue of every data sample in the vector. E.g. if a time series instant vectoris multiplied by 2, the result is another vector in which every sample value ofthe original vector is multiplied by 2.
Between two instant vectors, a binary arithmetic operator is applied toeach entry in the left-hand side vector and its matching elementin the right-hand vector. The result is propagated into the result vector with thegrouping labels becoming the output label set. The metric name is dropped. Entriesfor which no matching entry in the right-hand vector can be found are not part ofthe result.
Comparison binary operators
The following binary comparison operators exist in Prometheus:
==
(equal)!=
(not-equal)>
(greater-than)<
(less-than)>=
(greater-or-equal)<=
(less-or-equal)Comparison operators are defined between scalar/scalar, vector/scalar,and vector/vector value pairs. By default they filter. Their behavior can bemodified by providingbool
after the operator, which will return0
or1
for the value rather than filtering.
Between two scalars, the bool
modifier must be provided and theseoperators result in another scalar that is either 0
(false
) or 1
(true
), depending on the comparison result.
Between an instant vector and a scalar, these operators are applied to thevalue of every data sample in the vector, and vector elements between which thecomparison result is false
get dropped from the result vector. If the bool
modifier is provided, vector elements that would be dropped instead have the value0
and vector elements that would be kept have the value 1
.
Between two instant vectors, these operators behave as a filter by default,applied to matching entries. Vector elements for which the expression is nottrue or which do not find a match on the other side of the expression getdropped from the result, while the others are propagated into a result vectorwith the grouping labels becoming the output label set.If the bool
modifier is provided, vector elements that would have beendropped instead have the value 0
and vector elements that would be kept havethe value 1
, with the grouping labels again becoming the output label set.
Logical/set binary operators
These logical/set binary operators are only defined between instant vectors:
and
(intersection)or
(union)unless
(complement)vector1 and vector2
results in a vector consisting of the elements ofvector1
for which there are elements invector2
with exactly matchinglabel sets. Other elements are dropped. The metric name and values are carriedover from the left-hand side vector.
vector1 or vector2
results in a vector that contains all original elements(label sets + values) of vector1
and additionally all elements of vector2
which do not have matching label sets in vector1
.
vector1 unless vector2
results in a vector consisting of the elements ofvector1
for which there are no elements in vector2
with exactly matchinglabel sets. All matching elements in both vectors are dropped.
Vector matching
Operations between vectors attempt to find a matching element in the right-hand sidevector for each entry in the left-hand side. There are two basic types ofmatching behavior: One-to-one and many-to-one/one-to-many.
One-to-one vector matches
One-to-one finds a unique pair of entries from each side of the operation.In the default case, that is an operation following the format vector1 <operator> vector2
.Two entries match if they have the exact same set of labels and corresponding values.The ignoring
keyword allows ignoring certain labels when matching, while theon
keyword allows reducing the set of considered labels to a provided list:
<vector expr> <bin-op> ignoring(<label list>) <vector expr>
<vector expr> <bin-op> on(<label list>) <vector expr>
Example input:
method_code:http_errors:rate5m{method="get", code="500"} 24
method_code:http_errors:rate5m{method="get", code="404"} 30
method_code:http_errors:rate5m{method="put", code="501"} 3
method_code:http_errors:rate5m{method="post", code="500"} 6
method_code:http_errors:rate5m{method="post", code="404"} 21
method:http_requests:rate5m{method="get"} 600
method:http_requests:rate5m{method="del"} 34
method:http_requests:rate5m{method="post"} 120
Example query:
method_code:http_errors:rate5m{code="500"} / ignoring(code) method:http_requests:rate5m
This returns a result vector containing the fraction of HTTP requests with status codeof 500 for each method, as measured over the last 5 minutes. Without ignoring(code)
therewould have been no match as the metrics do not share the same set of labels.The entries with methods put
and del
have no match and will not show up in the result:
{method="get"} 0.04 // 24 / 600
{method="post"} 0.05 // 6 / 120
Many-to-one and one-to-many vector matches
Many-to-one and one-to-many matchings refer to the case where each vector element onthe "one"-side can match with multiple elements on the "many"-side. This has tobe explicitly requested using the group_left
or group_right
modifier, whereleft/right determines which vector has the higher cardinality.
<vector expr> <bin-op> ignoring(<label list>) group_left(<label list>) <vector expr>
<vector expr> <bin-op> ignoring(<label list>) group_right(<label list>) <vector expr>
<vector expr> <bin-op> on(<label list>) group_left(<label list>) <vector expr>
<vector expr> <bin-op> on(<label list>) group_right(<label list>) <vector expr>
The label list provided with the group modifier contains additional labels fromthe "one"-side to be included in the result metrics. For on
a label can onlyappear in one of the lists. Every time series of the result vector must beuniquely identifiable.
Grouping modifiers can only be used forcomparison andarithmetic. Operations as and
, unless
andor
operations match with all possible entries in the right vector bydefault.
Example query:
method_code:http_errors:rate5m / ignoring(code) group_left method:http_requests:rate5m
In this case the left vector contains more than one entry per method
labelvalue. Thus, we indicate this using group_left
. The elements from the rightside are now matched with multiple elements with the same method
label on theleft:
{method="get", code="500"} 0.04 // 24 / 600
{method="get", code="404"} 0.05 // 30 / 600
{method="post", code="500"} 0.05 // 6 / 120
{method="post", code="404"} 0.175 // 21 / 120
Many-to-one and one-to-many matching are advanced use cases that should be carefully considered.Often a proper use of ignoring(<labels>)
provides the desired outcome.
Aggregation operators
Prometheus supports the following built-in aggregation operators that can beused to aggregate the elements of a single instant vector, resulting in a newvector of fewer elements with aggregated values:
sum
(calculate sum over dimensions)min
(select minimum over dimensions)max
(select maximum over dimensions)avg
(calculate the average over dimensions)stddev
(calculate population standard deviation over dimensions)stdvar
(calculate population standard variance over dimensions)count
(count number of elements in the vector)count_values
(count number of elements with the same value)bottomk
(smallest k elements by sample value)topk
(largest k elements by sample value)quantile
(calculate φ-quantile (0 ≤ φ ≤ 1) over dimensions)These operators can either be used to aggregate over all label dimensionsor preserve distinct dimensions by including awithout
orby
clause.
<aggr-op>([parameter,] <vector expression>) [without|by (<label list>)]
parameter
is only required for count_values
, quantile
, topk
andbottomk
. without
removes the listed labels from the result vector, whileall other labels are preserved the output. by
does the opposite and dropslabels that are not listed in the by
clause, even if their label values areidentical between all elements of the vector.
count_values
outputs one time series per unique sample value. Each series hasan additional label. The name of that label is given by the aggregationparameter, and the label value is the unique sample value. The value of eachtime series is the number of times that sample value was present.
topk
and bottomk
are different from other aggregators in that a subset ofthe input samples, including the original labels, are returned in the resultvector. by
and without
are only used to bucket the input vector.
Example:
If the metric http_requests_total
had time series that fan out byapplication
, instance
, and group
labels, we could calculate the totalnumber of seen HTTP requests per application and group over all instances via:
sum(http_requests_total) without (instance)
Which is equivalent to:
sum(http_requests_total) by (application, group)
If we are just interested in the total of HTTP requests we have seen in allapplications, we could simply write:
sum(http_requests_total)
To count the number of binaries running each build version we could write:
count_values("version", build_version)
To get the 5 largest HTTP requests counts across all instances we could write:
topk(5, http_requests_total)
Binary operator precedence
The following list shows the precedence of binary operators in Prometheus, fromhighest to lowest.
^
*
,/
,%
+
,-
==
,!=
,<=
,<
,>=
,>
and
,unless
or
Operators on the same precedence level are left-associative. For example,2 3 % 2
is equivalent to(2
3) % 2
. However^
is right associative,so2 ^ 3 ^ 2
is equivalent to2 ^ (3 ^ 2)
.