LogQL: Log Query Language

Loki comes with its very own language for querying logs called LogQL. LogQLcan be considered a distributed grep with labels for filtering.

A basic LogQL query consists of two parts: the log stream selector and afilter expression. Due to Loki’s design, all LogQL queries are required tocontain a log stream selector.

The log stream selector will reduce the number of log streams to a manageablevolume. Depending how many labels you use to filter down the log streams willaffect the relative performance of the query’s execution. The filter expressionis then used to do a distributed grep over the retrieved log streams.

Log Stream Selector

The log stream selector determines which log streams should be included in yourquery. The stream selector is comprised of one or more key-value pairs, whereeach key is a log label and the value is that label’s value.

The log stream selector is written by wrapping the key-value pairs in apair of curly braces:

  1. {app="mysql",name="mysql-backup"}

In this example, log streams that have a label of app whose value is mysqland a label of name whose value is mysql-backup will be included in thequery results.

The = operator after the label name is a label matching operator. Thefollowing label matching operators are supported:

  • =: exactly equal.
  • !=: not equal.
  • =~: regex matches.
  • !~: regex does not match.

Examples:

  • {name=~"mysql.+"}
  • {name!~"mysql.+"}

The same rules that apply for Prometheus LabelSelectorsapply for Loki log stream selectors.

Filter Expression

After writing the log stream selector, the resulting set of logs can be filteredfurther with a search expression. The search expression can be just text orregex:

  • {job="mysql"} |= "error"
  • {name="kafka"} |~ "tsdb-ops.*io:2003"
  • {instance=~"kafka-[23]",name="kafka"} != kafka.server:type=ReplicaManager

In the previous examples, |=, |~, and != act as filter operators andthe following filter operators are supported:

  • |=: Log line contains string.
  • !=: Log line does not contain string.
  • |~: Log line matches regular expression.
  • !~: Log line does not match regular expression.

Filter operators can be chained and will sequentially filter down theexpression - resulting log lines must satisfy every filter:

{job="mysql"} |= "error" != "timeout"

When using |~ and !~,Go RE2 syntax regex may be used. Thematching is case-sensitive by default and can be switched to case-insensitiveprefixing the regex with (?i).

Counting logs

LogQL also supports functions that wrap a query and allow for counting entriesper stream.

Range Vector aggregation

LogQL shares the same range vectorconcept from Prometheus, except the selected range of samples include a value of1 for each log entry. An aggregation can be applied over the selected range totransform it into an instance vector.

The currently supported functions for operating over are:

  • rate: calculate the number of entries per second
  • count_over_time: counts the entries for each log stream within the givenrange.

count_over_time({job="mysql"}[5m])

This example counts all the log lines within the last five minutes for theMySQL job.

rate( ( {job="mysql"} |= "error" != "timeout)[10s] ) )

This example demonstrates that a fully LogQL query can be wrapped in theaggregation syntax, including filter expressions. This example gets theper-second rate of all non-timeout errors within the last ten seconds for theMySQL job.

Aggregation operators

Like PromQL,LogQL supports a subset of built-in aggregation operators that can be used toaggregate the element of a single vector, resulting in a new vector of fewerelements but with aggregated values:

  • sum: Calculate sum over labels
  • min: Select minimum over labels
  • max: Select maximum over labels
  • avg: Calculate the average over labels
  • stddev: Calculate the population standard deviation over labels
  • stdvar: Calculate the population standard variance over labels
  • count: Count number of elements in the vector
  • bottomk: Select smallest k elements by sample value
  • topk: Select largest k elements by sample value

The aggregation operators can either be used to aggregate over all labelvalues or a set of distinct label values by including a withou or aby clause:

<aggr-op>([parameter,] <vector expression>) [without|by (<label list>)]

parameter is only required when using topk and bottomk. topk andbottomk are different from other aggregators in that a subset of the inputsamples, including the original labels, are returned in the result vector. byand without are only used to group the input vector.

The without cause removes the listed labels from the resulting vector, keepingall others. The by clause does the opposite, dropping labels that are notlisted in the clause, even if their label values are identical between allelements of the vector.

Examples

Get the top 10 applications by the highest log throughput:

topk(10,sum(rate({region="us-east1"}[5m]) by (name))

Get the count of logs during the last five minutes, groupingby level:

sum(count_over_time({job="mysql"}[5m])) by (level)

Get the rate of HTTP GET requests from NGINX logs:

avg(rate(({job="nginx"} |= "GET")[10s])) by (region)